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Abstract 


HUMAN. A/S 


The  Coalition  Mission  Training  Research  (CMTR)  Computer-Supported,  Collaborative  Work 
(CSCW)  system  allows  geographically  separated  participants  to  collaborate  during  mission 
planning  and  briefmg/debriefmg.  Collaboration  within  the  CMTR  environment  is  focused  on 
facilitating  task-oriented  communication  among  team  members.  The  opportunity  was  taken  to 
conduct  a  video  analysis  and  questionnaire  survey  of  participants  in  a  distributed  briefing  and 
debriefing  environment  during  the  Maple  Skies  simulation  training  event.  The  video  analysis 
focused  on  behaviours  exhibited  by  participants  such  as  how  many  turns  they  had,  how  long  each 
phase  of  the  briefing  lasted,  collaborative  tool  use,  gestures,  and  interruptions.  The  questionnaire 
survey  solicited  a  participant’s  feelings  on  how  well  the  collaborative  tools  facilitated  the 
distributed  briefings  and  debriefings.  The  results  of  these  investigations  are  reported  and 
recommendations  for  future  development  of  this  work  are  made. 


Humansystems 


CSCW  Tool  Usage 


Page  i 


HU  MAN. 9  Y  STEMS 


Resume 


Le  systeme  de  travail  cooperatif  assiste  par  ordinateur  (CSCW)  des  recherches  sur  l'instruction  en 
vue  de  missions  de  la  coalition  (CMTR)  permet  aux  participants  eloignes  geographiquement  de 
collaborer  durant  la  planification  de  missions  et  le  breffage/  debreffage.  La  collaboration  en  milieu 
CMTR  met  l'accent  sur  les  moyens  de  faciliter  les  communications  orientees  sur  les  taches  entre  les 
membres  d'equipe.  II  s'est  presente  une  occasion  de  mener  une  analyse  video  et  un  sondage  au 
moyen  d'un  questionnaire  aupres  des  intervenants  dans  un  milieu  de  breffage  et  de  debreffage 
reparti  durant  l'exercice  destruction  de  simulation  Maple  Skies.  L'analyse  video  a  porte  sur  les 
comportements  manifestes  par  les  intervenants,  comme  le  nombre  de  fois  ou  ils  sont  intervenus,  la 
duree  de  chaque  phase  de  breffage,  l'utilisation  d'outils  de  collaboration,  les  gestes  et  les 
interruptions.  Le  sondage  par  questionnaire  demandait  aux  intervenants  a  quel  degre  les  outils  de 
collaboration  facilitaient  les  breffages  et  les  debreffages  repartis.  Les  resultats  des  enquetes  sont 
signales,  et  des  recommandations  sont  presentees  a  l'egard  de  la  poursuite  ulterieure  des  travaux. 
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Executive  Summary 


The  Simulation  and  Modelling  for  Acquisition,  Rehearsal  and  Training  (SMART)  group  at 
Defence  Research  and  Development  Canada  (DRDC)  Toronto  have  a  remit  to  research  distributed 
simulation  for  the  purposes  of  training.  Part  of  this  remit  is  the  consideration  of  computer 
supported  collaborative  working.  As  part  of  this  program  of  research,  a  simulation  (called  ‘Maple 
Skies’)  was  held  between  22nd  and  24th  February  2006.  This  simulation  was  originally  intended  to 
involve  the  US  Air  Force  and  other  Canadian  units  in  a  distributed  environment  that  would  involve 
flying  the  mission,  but  also  online  collaborative  briefing  and  debriefing.  To  facilitate  this,  the 
Coalition  Mission  Training  Research  (CMTR)  Computer-Supported  Collaborative  Work  (CSCW) 
system  allows  geographically  separated  participants  to  collaborate  during  mission  planning  and 
briefmg/debriefing.  Collaboration  within  the  CMTR  environment  is  focused  on  facilitating  task- 
oriented  communication  among  team  members.  Due  to  technical  difficulties,  the  other  participants 
withdrew  from  the  activity,  but  DRDC  Toronto  continued  with  the  simulation  and  created  a 
distributed  team  by  which  to  investigate  online  collaboration. 

Two  approaches  were  followed:  video  analysis  and  questionnaire  surveys.  The  questionnaire 
surveys  collected  data  about  the  participants’  level  of  comfort  with  common  computer  applications 
and  computer  use.  The  video  analysis  collected  data  concerning  one-to-many  and  many-to-many 
communication  measures.  One-to-many  communication  measures  included  overall  session  time, 
mission  briefing  and  debriefing  phase  times,  and  frequency  and  duration  of  technical  breakdowns. 
Many-to-many  communication  measures  included  CSCW  tools  used,  number  of  turns  for  each 
participant,  number  of  active  participants,  transitions  between  participants,  and  incidence  of 
overlaps  between  participants.  In  total  there  were  up  to  10  participants,  although  only  four 
completed  the  surveys. 

Analysis  showed  that  the  mission  briefing  and  debriefing  format  were  rigidly  structured  and 
controlled  by  the  mission  commander.  There  was  little  interruption  or  questioning  by  other 
participants.  The  mission  commander  had  the  most  turns  and  the  longest  speaking  durations.  Few 
technical  breakdowns  were  observed  and  all  participants  replied  that  they  were  comfortable  with 
using  computers. 

The  investigation  was  observational  in  nature  and  thus  no  control  was  exerted  over  the  situations  in 
which  participants  found  themselves.  This  resulted  in  data  from  which  it  is  difficult  to  draw 
conclusions.  Nevertheless,  this  investigation  did  result  in  a  number  of  recommendations  regarding 
how  the  investigation  could  result  in  more  information  outcomes  if  performed  again.  What 
conclusions  could  be  drawn  and  the  recommendations  for  future  work  are  discussed. 
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Sommaire 


Le  groupe  de  simulation  et  de  modelisation  pour  l'acquisition,  la  repetition  et  l'entrainement 
(SMARE)  de  Recherche  et  Developpement  pour  la  defense  Canada  (RDDC)  Toronto  a  le  mandat 
de  mener  des  recherches  sur  la  simulation  repartie  aux  fins  destruction.  Ce  mandat  comprend 
notamment  l'examen  du  travail  cooperatif  assiste  par  ordinateur  (CSCW).  Dans  le  cadre  du 
programme  de  recherche,  un  exercice  de  simulation  (appele  «  Maple  Skies  »)  a  eu  lieu  du 
22  au  24  fevrier  2006.  L'exercice  reposait  au  depart  sur  la  participation  de  la  US  Air  Force  et 
d'unites  du  Canada  dans  un  milieu  reparti,  qui  supposait  l'execution  de  la  mission  en  vol,  mais  aussi 
la  tenue  de  breffage  et  de  debreffage  en  ligne.  A  cette  fin,  le  systeme  CSCW  des  recherches  sur 
l'instruction  en  vue  de  missions  de  la  coalition  (CMTR)  permet  aux  participants  eloignes 
geographiquement  de  collaborer  durant  la  planification  de  missions  et  le  breffage/debreffage.  La 
collaboration  en  milieu  CMTR  met  l'accent  sur  les  moyens  de  faciliter  les  communications 
orientees  sur  les  taches  entre  les  membres  d'equipe.  A  cause  de  difficultes  techniques,  les  autres 
participants  se  sont  retires  de  l'activite,  mais  RDDC  Toronto  a  poursuivi  la  simulation  et  cree  une 
equipe  repartie  chargee  d'etudier  la  collaboration  en  ligne. 

Deux  approches  ont  ete  adoptees  :  l'analyse  video  et  un  sondage  au  moyen  d'un  questionnaire.  Le 
sondage  par  questionnaire  a  permis  de  recueillir  des  donnees  sur  le  niveau  d'aisance  des 
participants  avec  des  applications  informatiques  courantes  et  l'utilisation  d'un  ordinateur.  L'analyse 
video  a  permis  de  recueillir  des  donnees  sur  les  communications  d'un  a  plusieurs  et  de  plusieurs  a 
plusieurs.  Les  donnees  de  communications  d'un  a  plusieurs  comprenaient  la  duree  globale  des 
seances,  la  duree  des  phases  de  breffage  et  de  debreffage  de  la  mission,  ainsi  que  la  duree  et  la 
frequence  des  avaries  techniques.  Les  donnees  de  communications  de  plusieurs  a  plusieurs 
comprenaient  les  outils  CSCW  utilises,  le  nombre  d'interventions  de  chaque  participant,  le  nombre 
de  participants  actifs,  les  transitions  entre  les  participants  et  l'incidence  des  chevauchements  entre 
les  participants.  En  tout,  on  a  compte  jusqu'a  dix  participants,  meme  si  seulement  quatre  ont  pris 
part  au  sondage. 

L'analyse  a  montre  que  le  breffage  et  le  debreffage  de  la  mission  etaient  structures  selon  un  format 
rigide,  dirige  par  le  commandant  de  mission.  II  y  a  eu  peu  d'interruptions  ou  de  questions  des  autres 
participants.  C'est  le  commandant  de  mission  qui  a  fait  le  plus  d'interventions  et  avait  la  parole  le 
plus  souvent.  Peu  d'avaries  techniques  ont  ete  observees,  et  tous  les  participants  ont  indique  se 
sentir  a  l'aise  a  utiliser  un  ordinateur. 

L'enquete  reposait  sur  des  observations,  ce  qui  veut  dire  qu'aucun  controle  n'a  ete  exerce  sur  les 
situations  dans  lesquelles  les  participants  se  sont  trouves.  Cela  explique  qu'on  a  obtenu  des  donnees 
a  partir  desquelles  il  est  difficile  de  tirer  des  conclusions.  Cette  enquete  s'est  neanmoins  traduite  par 
un  certain  nombre  de  recommandations  sur  la  fa9on  dont  l'enquete  pourrait  permettre  d'obtenir  plus 
d'information  si  elle  etait  effectuee  de  nouveau.  Les  conclusions  qui  pourraient  etre  tirees  et  les 
recommandations  concemant  des  recherches  futures  sont  examinees. 
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The  Coalition  Mission  Training  Research  (CMTR)  Computer- Supported,  Collaborative  Work 
(CSCW)  system  allows  geographically  separated  participants  to  collaborate  during  mission 
planning  and  briefmg/debriefmg.  Collaboration  within  the  CMTR  environment  is  focused  on 
facilitating  task-oriented  communication  among  team  members. 

The  CSCW  system  in  the  CMTR  boardroom  is  a  collection  of  hardware  and  software  components 
that  allow  team  members  to  present  and  share  information  in  a  timely  manner.  The  components  of 
the  CMTR  CSCW  system  are  loosely  coupled,  with  each  component  only  providing  a  small  subset 
of  the  required  features/functions  (some  overlap  between  components  and  features  exists).  This 
definition  differs  from  most  of  the  CSCW  systems  (and  related  evaluations)  found  in  the  literature 
in  that  those  systems  are  typically  tightly  coupled  and  in  many  cases,  a  single  hardware/software 
package  implements  all  of  the  features  being  evaluated.  A  federated  CSCW  system  has  the 
disadvantage  that  information  and  awareness  are  not  easily  shared  or  obtained  between  the 
components  and  there  is  a  heavy  reliance  on  the  user  to  maintain  a  consistent  situational  and  social 
awareness  when  switching  between  applications.  Alternatively,  the  federated  system  approach  has 
the  advantage  that  components  can  easily  be  added  or  removed  to  tailor  the  overall  feature  set  of 
the  system  in  order  to  meet  the  team’s  communication  and  coordination  (collaboration)  needs. 

The  CMTR  CSCW  system  at  Defence  Research  and  Development  Canada  (DRDC)  Toronto 
currently  consists  of  the  following  components: 

1.  SMART  Board  Interactive  Whiteboard  (www.smarttech.com) 

2.  Voice  conferencing  system 

a.  VoIP  Phones 

3.  Video  conferencing  system 

a.  Polycom  (www.Polycom.com) 

4.  Desktop  computers  -  Windows  NT/2000/XP 

5.  Desktop  applications: 

a.  PowerPoint 

b.  Falcon  View  -  flight  plan  viewer 

c.  DCS  -  mission  log  viewer 

The  primary  role  of  the  CMTR  CSCW  system  is  to  support  collaboration  between  team  members 
(pilots,  ATC  officers,  etc)  in  a  simulated  air  force  mission  environment.  There  are  three  high-level 
stages  of  an  air  force  mission:  planning,  execution,  and  debriefing.  The  planning  and  debriefing 
stages  are  of  interest  because  of  the  possible  use  of  a  CSCW  system  within  these  stages. 

The  planning  stage  can  be  divided  into  the  following  sub-stages:  mission  tasking,  detailed  flight 
plan  development,  and  coordination  briefing.  During  the  mission  tasking  the  overall  mission  plan 
is  presented  to  all  participants  (also  referred  to  as  the  Overview  Briefing).  During  the  flight  plan 
development,  individual  participants  are  responsible  for  producing  a  flight  plan  that  is  constrained 
by  the  overall  mission  briefing  and  local  factors  (e.g.,  weather,  war  zones,  and  aircraft  limitations). 
Once  the  individual  flight  plans  are  completed,  a  final  briefing  is  presented  which  combines  the 
individual  flight  plans.  After  this  stage,  all  participants  are  appropriately  tasked.  Once  the  briefing 
is  completed,  the  mission  is  flown  (mission  execution).  Following  the  mission  execution  is  the 
debriefing  stage.  A  key  result  of  the  mission  debriefing  is  the  identification  of  lessons  learned. 

The  debriefing  can  be  either  a  structured  or  an  unstructured  discussion  of  the  team  members’ 
impressions  of  the  mission  execution.  The  discussion  highlights  both  negative  and  positive  aspects 
of  the  mission  execution. 


Y\umwsystems  Incorporated 


CSCW  Tool  Usage 


Page  1 


HUMANS  KSTf  MS 

l*C9rPor*t<4 


Therefore,  the  types  of  communication  and  coordination  that  the  CMTR  CSCW  system  must 
accommodate  are  diverse  and  the  use  of  a  federation  of  components  may  have  merit  in  terms  of 
supporting  the  communication  needs  of  the  team  members  but  typically  at  the  cost  of  coordination 
(distributed  process  loss). 

Past  research  regarding  the  effectiveness  of  the  CMTR  CSCW  system  has  primarily  focused  on 
Quality  of  Service  and  functional  requirements  issues.  This  has  been  accomplished  by  reviewing 
relevant  data  logs  and  by  interviewing  users  after  a  session  to  elicit  comments  about  the  user’s 
satisfaction  of  the  various  communication  and  collaboration  tools.  These  results  were  than  used  as 
anecdotal  evidence  as  to  which  tools  should  be  replaced  for  future  sessions. 

These  past  studies  have  lead  to  a  CSCW  system  that  seems  to  meet  the  needs  of  the  users  but  it  is 
unclear  as  to  what  those  needs  are.  Therefore,  one  of  the  primary  motivations  for  this  research  is  to 
understand  the  needs  of  the  users  of  this  CSCW  environment.  Along  with  understanding  the  user 
requirements,  it  is  a  goal  of  this  research  to  produce  a  repository  of  collaboration  patterns  that  can 
be  used  to  analyze  and  evaluate  other  collaborative  systems  for  military  applications  (e.g.,  mission 
planning,  briefing,  and  debriefing).  This  repository  of  patterns  is  of  particular  importance  as  the 
Canadian  Forces  are  a  geographic  dispersed  military  that  require  timely  access  to  information  thus 
the  increasing  importance  on  CSCW  and  groupware  technologies.  As  the  reliance  on  such 
technologies  increases,  the  military  will  need  an  efficient  means  (selection  criteria)  to  assist  in  the 
selection  of  various  tools  for  different  application  environments.  In  addition  to  identifying  patterns 
that  need  to  be  supported  by  a  CSCW  application,  the  tasks  encapsulated  in  those  patterns  can  be 
linked  back  to  skills  and  training  requirements  in  order  to  successfully  perform/operate/work  with 
the  CSCW  application. 

A  planned  CMTR  distributed  simulation  event,  Maple  Skies,  was  used  to  collect  data  on  CSCW 
collaboration  patterns.  This  simulation  took  place  in  the  Simulated  Environments  Research 
Facility  (SERF)  at  DRDC  Toronto,  and  used  the  Multi-Task  Trainer  (MTT;  a  CF-18  simulator)  and 
the  associated  Next  Generation  Threat  Simulator  (NGTS).  Data  was  collected  on  video  tape  and 
via  behavioural  surveys.  All  data  was  subject  to  offline  analysis  once  the  CMTR  event  was 
concluded. 

This  work  was  performed  under  contract  W771 1-03787 1//00 1/TOR,  call-up  7871-07.  The 
Scientific  Authority  (SA)  for  this  work  was  Dr.  Rick  Bodner. 
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There  were  three  main  elements  to  this  work:  video  data  collection;  behavioural  survey  data 
collection;  and  data  analysis.  Each  element  will  be  discussed  separately  below. 


2.1  Video  Data  Collection 

The  planning,  briefing,  and  debriefing  activities  were  conducted  in  two  rooms,  both  within  the 
SERF,  but  located  on  different  floors.  The  room  normally  used  for  flight  planning  and  video 
conferences  was  downstairs.  Upstairs,  a  shared  office  had  been  commandeered  for  the  purposes  of 
distributed  planning.  Because  most  collaborative  tools  were  located  downstairs,  the  focus  of  video 
data  collection  was  downstairs.  Upstairs,  only  one  video  camera  was  used  to  capture  the  whole 
room  and  all  occupants.  Downstairs,  three  video  cameras  were  arranged  in  three  corners  of  the 
room,  supplemented  by  external  microphones.  This  coverage  was  to  ensure  all  behaviours  were 
captured  (see  Figure  1).  Additionally,  the  Polycom  videoconferencing  system  was  recorded  to 
video  tape  to  ensure  that  a  ‘primary’  view  was  captured  of  both  the  downstairs  and  upstairs. 
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Figure  1:  Room  Layout,  Typical  Participant  Positions  and  Data  capture 

Configuration 
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The  video  capture  devices  used  to  collect  the  video  data  are  presented  in  Table  1. 

Table  1  :  Video  Capture  Equipment 


Item 

Quantity 

VIDEO  CAMERAS 

4 

Sony  TRV87  8mm  camcorder  (camera  2) 

1 

Sony  TRV240  Digital  camcorder  (cameras  3  and  4) 

2 

Sony  TRV110  Digital  camcorder  (camera  1) 

1 

RECORDER  DECKS 

2 

JVC  S-Video  recorder  (used  with  camera  3) 

1 

Sony  4-Head  VCR  (used  with  Polycom  system) 

1 

VIDEO  CONFERENCE  SYSTEM 

1 

Polycom 

1 

MICROPHONES 

2 

Sony  ECM-MS908C  (used  with  cameras  1  and  2) 

2 

The  CMTR  event  ran  from  Wednesday  22nd  February  to  Friday  24th  February,  2006.  Each  day 
consisted  of  at  least  one  briefing  and  one  debriefing.  On  some  days,  more  than  one  briefing  was 
captured.  After  each  briefing  and  debriefing  period,  the  video  tape  in  each  camera  was  replaced. 
All  tapes  were  returned  to  DRDC  Toronto  for  safe-keeping  or  destruction  at  the  conclusion  of  this 
work. 

2.2  Behavioural  Survey  Data  Collection 

Initially,  the  participants  in  this  study  were  to  be  the  CMTR  event  training  audience  (i.e.  CF-18 
pilots  and  airborne  weapons  controllers)  and  associated  Subject  Matter  Experts  (SMEs).  Each 
participant  was  given  a  computer-experience  questionnaire  and  a  performance  measurement 
questionnaire  after  each  briefing  and  each  debriefing.  After  review  of  these  performance 
measurement  questionnaires,  it  was  decided  to  amalgamate  them  with  the  broader  Measures  of 
Performance  (MOPs)  questionnaire  focusing  on  training  effectiveness.  The  original  questionnaire 
items  are  shown  in  Annex  A-4  through  to  Annex  A-25.  The  resultant  questionnaire  items  are 
shown  in  Annex  A-2  and  Annex  A-3.  In  total,  six  participants  filled  in  behavioural  surveys:  two 
CF-18  pilots,  two  airborne  weapons  controllers,  and  two  SMEs  (both  former  CF-18  pilots). 

2.3  Data  Analysis 

Upon  conclusion  of  the  CMTR  event,  the  video  taped  data  was  transferred  digitally  to  a  non-linear 
editing  system.  The  non-linear  editing  software  application  provided  the  means  to  obtain  a 
synchronous  time  code  across  the  five  different  recording  devices  and  provide  a  process  to 
accurately  time  stamp  specific  communication  measures.  The  tapes  were  then  reviewed  and 
instances  of  particular  behaviours  were  noted. 

Additionally,  relevant  details  were  captured  whenever  there  was  a  shift  or  change  in  the  state  of  the 
session  (e.g.,  pilots  leave  briefing  room  since  their  section  of  the  brief  has  ended,  group  motivation 
waning  do  to  technical  difficulties,  etc.).  Behavioural  survey  responses  were  used  to  supplement 
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the  data  captured  from  the  video  record.  With  a  small  subject  sample,  it  is  not  possible  to  do 
statistical  testing  on  the  survey  data  beyond  simple  descriptive  statistics.  The  results  of  this  data 
compilation  will  be  reported  in  the  next  section. 

2.4  Behaviour  Notes 

2.4.1  Social  Parameters 

The  chair  of  the  mission  maintained  loose  control  with  a  pre- determined  agenda  open  to 
modification.  No  rules  of  order  were  used,  but  floor  control  was  dictated  by  the  agenda  and  a 
general  understanding  the  chair  (or  moderator)  was  responsible  for  introducing  each  participant 
prior  to  each  phase  presentation.  No  official  rank  titles  were  used  among  the  participants  other  than 
their  call  signs.  All  communication  consisted  of  a  combination  of  one-to-many  and  many-to-many 
exchanges  and  was  public. 

2.4.2  Group  Characteristics 

The  group  involved  in  the  CSCW  tool  usage  study  consisted  of  the  following  participants:  2 
airborne  weapons  controllers,  2  CF-18  pilots,  4  retired  air  force  captains.  Figure  2  represents  the 
hierarchical  breakdown  and  military  status  of  the  individuals  involved  with  the  study.  Originally, 
the  Canadian  Forces  Base  (CFB)  Bagotville  and  Air  Force  Research  Laboratory  (AFRL)  Mesa 
were  to  participate  in  Maple  Skies,  but  due  to  technical  issues  they  were  not  involved. 


P 2  -  Lt.  Colonel 
(retired) 


PI  -  Major 


P 3  -  Captain 
(retired) 


P4  -  Captain 

P5  -  Captain 

(retired) 

P6  -  Captain 
(retired) 


P8  -  Captain 


P11  -  Captain 


PW  Lieutenant 


P7  -  Technical  Support 


Figure  2:  Hierarchical  Representation  of  Rank  of  Participants 

The  number  of  members  in  total  was  1 1  including  one  technical  support  person.  The  number  of 
team  members  per  location  was  not  fixed  and  varied  per  mission  and  session.  Two  locations 
were  used,  both  located  in  DRDC’s  Toronto  SERF  area.  One  room  was  located  downstairs  and 
one  upstairs  (refer  to  Figure  1.0  for  room  configuration).  The  group  as  a  whole  was  newly 
formed,  but  the  majority  of  the  members  were  familiar  with  one  another  from  previous 
engagements.  The  time  allotted  for  each  mission  brief  and  debrief  was  to  a  maximum  of  45 
minutes. 
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2.4.3  Technology  Characteristics 

The  groupware  (hardware  and  software)  available  for  the  CSCW  tool  usage  study  at  DRDC 
Toronto  included: 

•  SMART  Board  Interactive  Whiteboard  (www.smarttech.com) 

•  Voice  conferencing  system  -  VoIP  Phones 

•  Video  conferencing  system  -  Polycom  (www.polycom.com) 

•  Desktop  computers  -  Windows  NT/2000/XP 

•  Desktop  applications  included: 

o  Microsoft  Office  Suite  (w/Power  Point) 
o  Falcon  View  -  flight  plan  viewer 
o  DCS  -  mission  log  viewer 

The  downstairs  SERF  room  contained  the  SMART  Board  Interactive  Whiteboard;  a  Polycom 
system;  telephones;  various  desktop  computers  workstations;  a  presentation  operator  station; 
Microsoft  Office  applications;  falcon  view;  and  the  DCS  mission  log  viewer.  The  upstairs  SERF 
room  included  a  Polycom  system;  telephone;  and  2  desktop  computers  with  Microsoft  Office.  The 
downstairs  SERF  room  employed  a  technology  operator  and  the  upstairs  SERF  room  did  not. 

The  minimum  hardware  requirements  for  a  site  to  join  the  conference  are  a  telephone  line  and 
telephone  (dial-in  to  join  the  conference).  Based  on  the  minimum  hardware  requirements,  no 
software  is  required. 
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3.1  Measures  of  Performance 

Due  to  the  small  sample  size,  only  descriptive  statistical  techniques  were  used.  The  results  of  the 
Measures  of  Performance  (MOP)  data  (see  Figure  3;  see  pages  A-4  -  A-25  for  the  content  of  each 
MOP)  shows  a  trend  toward  higher  ratings  (increased  number  of  4/5  ratings  and  higher  overall 
averages)  later  in  the  exercise  compared  to  early  in  the  exercise.  The  Generic  MOPs  increased  on 
average  from  3.5  to  4.67  (greater  than  a  1  point  increase).  The  higher  ratings  imply  the  quality  of 
the  briefing,  execution  and  debriefing  increased  as  the  exercise  progressed. 

Possible  reasons  for  the  increase  in  quality  observed  by  the  SMEs  could  be  attributed  to  the 
following  learning  effects: 

■  increased  comfort  levels  in  the  DRDC  Toronto  environment 

■  increased  productivity  through  familiarity  of  the  individuals  working  together 

■  adaptation  to  the  daily  schedule 

■  perceived  increase  in  quality  by  the  SMEs  when  in  fact  there  was  no  change 

The  Generic  MOPs  used  behaviourally  anchored  rating  scales  (BARS)  for  each  characteristic.  Due 
to  the  relatively  high  ratings,  it  should  be  determined  whether  the  pilots  truly  performed  at  the 
reported  level  or  if  the  behavioural  anchors  affected  the  rating.  If  the  latter  is  the  case,  recalibration 
of  the  behavioural  anchors  may  be  required. 


Figure  3:  Daily  Generic  MOPs  -  Briefing 
3.2  Questionnaires 

The  computer  experience  questionnaire  (see  Annex  A-l)  and  the  CSCW  survey  (see  Annex  A-2) 
were  distributed  to  both  the  CF-18  pilots  (P4  and  PI)  and  the  airborne  weapons  controllers  (PI  1 
and  P8). 
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3.2.1  Computer  Experience 

The  results  of  the  questionnaire  are  presented  in  Table  2:  below.  Due  to  the  small  sample  size 
(n=4),  no  statistical  analyses  were  conducted.  All  the  Computer  Experience  Questionnaire  (CEQ) 
recipients  had  comparable  computer  experience  reporting  daily  usage  of  computers  and  the 
Microsoft  Office  Suite  software  package.  The  majority  of  the  participants  stated  high  comfort 
levels  with  Microsoft  Office  software. 


Table  2:  Computer  Experience 


Question 

AWACS 

PILOTS 

Pll 

P8 

P4 

PI 

CEQ  1 

daily 

daily 

daily 

daily 

CEQ  2 

daily 

daily 

daily 

daily 

CEQ  3 

very  comfortable 

very  comfortable 

somewhat  comfortable 

daily 

3.2.2  Computer-Supported,  Collaborative  Work  Survey 

The  results  of  the  CSCW  survey  are  shown  in  Table  3  (see  pages  A-2  -  A-3  for  details  of  each 
question1).  All  respondents  responded  identically  to  the  survey  items  with  the  exception  of  CB2 
and  CD2.  Participants  did  not  complete  CSCW  surveys  for  the  following  mission  sessions:  Mission 
1  Initial  Brief,  Mission  2  mid  and  final  coordination  briefs,  and  Mission  2  Striker/Transport  brief. 


Table  3:  Computer-Supported,  Collaborative  Work  survey 


AWACS 

PILOTS 

Day 

BID 

Question 

Pll 

P8 

P4 

PI 

Wednesday 

Briefing 

CB1 

y 

y 

y 

y 

CB1  Note 

CB2 

y 

y 

y 

n 

CB2  Note 

CB3 

y 

y 

y 

y 

CB3  Note 

CB4 

y 

y 

y 

y 

CB4  Note 

CB5 

y 

y 

y 

NA 

CB5  Note 

Their  title  slide 
came  up  as  was 
pointed  out  by 
package  comd 

PowerPoint  pres. 

Was  in  our  room  and 
easy  to  follow 

CB6 

n 

n 

n 

y 

CB6  Note 

only  after  the  brief  in 
order  to  clarify  finer 
points 

possibly  for 

technical 

coordinators 

You  want  everyone 
listening  to  the 
briefer  so  they  don't 
miss  anything 

Debriefing 

CD1 

y 

y 

y 

n 

CD1 

Note 

lack  of  lessons 
learned  due  to 
lack  of  realism 

1  Note  that  some  participants  entered  responses  in  the  ‘notes’  section  associated  with  questions.  Sometimes 
these  are  meant  to  qualify  a  ‘level’  associated  with  their  response,  while  other  times  it  seems  they  were 
attempting  to  complete  the  questionnaire  quickly  and  may  have  made  a  mistake. 
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3.3  One-to-Many  Communication  Measures 

3.3.1  Overall  Session  Time 

Each  mission  consisted  of  at  least  one  brief  and  one  debrief  session.  The  total  duration  for  each 
session  was  recorded  and  reported  in  minutes.  Overall,  the  majority  of  the  briefing  and  debriefing 
sittings  for  missions  1  through  3  averaged  18.2  minutes  (see  Table  4).  Two  sessions  that  differed 
from  the  average  were  the  Mass  Brief  from  Mission  1  and  the  Strikers/Transport  Debrief  from 
Mission  2  with  durations  of  29.1  minutes  and  8.6  minutes,  respectively.  The  video  analysis  data 
suggest  that  the  longer  length  of  the  Mass  Brief  in  Mission  1  was  a  result  of  poor  integration  and 
time  intensive  configuration  of  the  DCS  mission  log  viewer  into  the  CSCW  system;  the  shorter 
length  of  the  Strikers/Transport  Debrief  was  a  result  of  not  involving  all  the  participants  directly. 

Table  4:  Overall  Session  Time 


Mission 

Session 

Duration  (minutes) 

1 

Initial  Brief 

20.5 

Mass  Brief 

20.8 

Mass  Debrief 

29.1 

2 

Mid  Coordination  Brief 

14.1 

Final  Coordination  Brief 

12.8 

Mass  Brief 

20.8 

Strikers/Transport  Debrief 

8.6 

Mass  Debrief 

19.8 

3 

Mass  Brief 

17.7 

Average  across  sessions 

18.2 

3.3.2  Mission  Phase  Time 

Missions  1  to  3  had  comparable  average  lengths  within  the  Mass  Brief  (see  Table  5).  Transition 
time  between  phases  for  missions  1  to  3  had  slight  variability.  The  “role  call”  phase  in  mission  1 
was  much  longer  duration  compared  to  the  others.  The  reason  the  role  call  phase  was  longer  in 
mission  1  was  because  role  call  was  conducted  on  a  per  role  basis.  Subsequent  role  call  phases 
consisted  of  “is  everyone  here?”  questions  resulting  in  both  distributed  teams  answering  “yes” 
together. 


Table  5:  Mass  Brief  -  Mission  1,  2,  and  3 


Mission  Phase 

Mission  1 

Mission  2 

Mission  3 

Length 

(sec) 

Transition 
Time  (sec) 

Length 

(sec) 

Transition 
Time  (sec) 

Length 

(sec) 

Transition 
Time  (sec) 

Welcome  -  Introduction 

13.1 

0.0 

9.2 

0.0 

6.06 

0 

Role  Call 

81.2 

1.0 

8.0 

0.0 

5.51 

0 

Weather 

167.1 

7.6 

66.3 

4.0 

31.12 

0 

Training  Rules 

125.4 

2.3 

77.9 

1.3 

72.06 

0.19 

Intelligence  brief 

43.6 

1.3 

45.2 

4.2 

NA 

NA 

Package  Commander 

84.9 

1.0 

205.3 

0.0 

196 

10.27 

CGR 

NA 

NA 

NA 

NA 

105 

1 

AWACS 

45.2 

3.2 

65.0 

4.5 

37.3 

0.91 

OCA  Commander 

56.4 

3.8 

45.2 

1.0 

61.76 

1.76 

SEAD 

43.0 

2.5 

44.3 

27.7 

51.27 

5.21 

DT 

30.6 

2.3 

21.5 

9.2 

NA 

NA 

Transport 

20.1 

9.1 

64.2 

0.9 

30 

3.61 
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Mission  Phase 

Mission  1 

Mission  2 

Mission  3 

Length 

(sec) 

Transition 
Time  (sec) 

Length 

(sec) 

Transition 
Time  (sec) 

Length 

(sec) 

Transition 
Time  (sec) 

Hold  Plan 

24.3 

5.6 

11.0 

0.0 

27.36 

1.79 

Rehearsal 

231.9 

10.3 

257.0 

2.5 

252.91 

0 

Contingency 

32.8 

0.7 

212.0 

13.2 

134.27 

0.18 

MISMO  -  Summary 

196.5 

45.5 

26.76 

Average  Time 

79.7 

3.6 

78.5 

4.9 

74.1 

1.9 

Mission  1  phase  transition  time  was  twice  the  length  compared  to  the  phase  transition  time  of 
Mission  2  within  the  Mass  Debrief  (see  Table  6).  The  difference  in  transition  time  between  phases 
can  be  attributed  to  the  “Route  Playback  -  DCS”  phase.  The  “Route  Playback  -  DCS”  in  Mission  1 
was  higher  than  Mission  2  because  of  technical  issues  with  the  DCS  software. 


Table  6:  Mass  Debrief  Mission  1  and  2 


Mission  1 

Mission  2 

Length 

Transition 

Length 

Transition 

Mission  Phase 

(sec) 

Time  (sec) 

(sec) 

Time  (sec) 

Welcome  -  Introduction 

12.1 

0.7 

14.4 

0.0 

Role  Call 

12.6 

0.0 

6.0 

0.0 

Debrief  ROE 

15.3 

* — 1 

o 

11.4 

2.9 

Package  Commander 

16.9 

0.0 

6.2 

0.0 

Game  Plan  Overview 

28.9 

0.0 

12.3 

0.0 

Air  to  Air  Commander 

27.5 

2.2 

81.1 

1.8 

SEAD 

52.8 

0.0 

21.2 

0.0 

AWACS 

NA 

NA 

4.9 

0.0 

DT 

NA 

NA 

47.8 

0.0 

Transport 

27.7 

1.8 

41.0 

9.5 

AWACS 

12.2 

5.3 

14.2 

4.9 

Route  Playback  -  DCS 

939.9 

76.0 

498.5 

15.6 

OCA 

58.3 

0.0 

151.7 

3.9 

SEAD 

42.0 

0.0 

47.3 

3.7 

SEAD 

28.7 

0.0 

NA 

NA 

Transport 

28.5 

4.9 

17.9 

3.4 

GCI/AWACS 

147.3 

4.1 

14.4 

3.1 

Package  Commander  Summary 

85.2 

0.0 

126.8 

1.7 

MISMO  Summary 

112.6 

23.0 

Average  Time 

97.0 

5.9 

63.3 

3.0 

Phase  data  for  Mission  1  Initial  Brief,  Mission  2  Mid  and  Final  Coordination  Briefs,  and  Mission  2 
Strikers/Transport  Debrief  are  reported  in  Table  7,  Table  8,  Table  9,  and  Table  10  respectively. 
These  sessions  varied  in  phase  composition  and  order  of  presentation. 
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Table  7:  Mission  1  -  Initial  Brief  Phase  Duration 


Phase 

Total  (sec) 

Time  Between 
Phase  (sec) 

Description  of  Work 

1067.5 

0 

Questions 

161.6 

Average  Time 

614.6 

0 

Table  8:  Mission  2  -  Mid  Coordination  Brief 


Phase 

Total  (sec) 

Time  Between 
Phase  (sec) 

Role  Call 

2.4 

10.6 

Review  Overview 

37.4 

1.0 

Planning  ROE 

7.1 

2.4 

Game  Plan 

45.1 

1.7 

Hold  Plan 

52.0 

0.8 

Game  Plan  Overview 

170.9 

0.6 

Air  to  Air  Game  Plan 

77.7 

23.9 

Transport 

22.6 

5.9 

SEAD 

86.5 

2.2 

DT 

69.6 

2.3 

Tasking 

204.8 

0.0 

Questions 

20.8 

Average  Time 

66.4 

4.7 

Table  9:  Mission  2  -  Final  Coordination  Brief 


Phase 

Total  (sec) 

Time  Between 
Phase  (sec) 

Start 

2.7 

0.0 

Role  Call 

4.0 

2.4 

Changes  to  MidCord 

22.0 

0.0 

Route  Rehearsal 

576.0 

19.1 

Whiteboard  (excel  spreadsheet) 

66.2 

10.8 

Comms  Card 

9.3 

0.0 

Summary 

29.1 

1.3 

Questions 

26.4 

Average  Time 

92.0 

4.8 
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Table  10:  Mission  2  -  Strikers  and  Transport  Debrief 


Phase 

Total  (sec) 

Time  Between 
Phase  (sec) 

Start 

13.8 

0.0 

MISMO 

2.9 

4.2 

Roll  Call 

8.7 

0.5 

Mission  Objectives 

20.4 

0.0 

Game  Plan  Overview 

36.6 

1.7 

AWACS 

17.4 

0.6 

Individual  Flight  Reviews 

51.1 

0.0 

Strikers 

68.2 

2.8 

Transport 

17.8 

0.0 

Lessons  Learned 

72.1 

0.0 

OCA  Execution 

11.5 

0.1 

Slide  preparation  for  Mass  Brief 

21.0 

0.0 

Prepare  DCS  for  Mass  Brief 

163.7 

Average  Time 

38.7 

0.8 

3.3.3  Breakdown  Time 

The  total  numbers  of  breakdowns  are  presented  in  Table  11.  Overall,  there  were  few 
breakdowns  noted.  Of  the  breakdowns  encountered,  few  were  longer  than  one  minute. 


Table  1 1 :  Number  of  Breakdowns 


Session 

Mission 

Total 

1 

2 

3 

FINAL  CORD 

1 

1 

INTB 

1 

1 

MB 

2 

1 

1 

4 

MD 

3 

2 

5 

MIDCORD 

2 

2 

STMD 

1 

1 

Total 

6 

7 

1 

14 

The  average  breakdown  length  was  approximately  90  seconds.  A  significant  software  breakdown 
occurred  in  Mission  1  during  the  Mass  Debrief.  After  further  investigation,  Table  6  (phase  data) 
reveals  the  software  breakdown  was  directly  related  to  the  DCS  mission  log  viewer  by  comparing 
the  phase  and  breakdown  durations  (Table  12).  The  highest  numbers  of  breakdowns  were  a  result 
of  user  error.  This  was  somewhat  unexpected  given  the  high  comfort  and  experience  level  reported 
by  participants  in  Table  2.  No  hardware  errors  occurred.  The  acronym  LOE  stands  for  “lack  of 
operating  experience”.  LOE  identifies  instances  when  the  computer  operator  does  not  have  the 
knowledge  or  experience  to  execute  a  software  application  command.  LOE  was  resolved  either  by 
the  on  hand  technical  support  person  (P7)  or  by  a  team  member.  “U”  identifies  instances  of  user 
error.  User  error  occurs  when  the  computer  operator  mistakenly  executed  an  incorrect  command  or 
pressed  the  wrong  key/mouse  click  causing  a  delay  in  the  presentation.  User  error  extended  to  all 
participants  to  cover  accidental  errors  (e.g.,  touching  SMART  board  causes  operator  to  loose 
control  of  presentation).  “S”  represents  the  failure  of  a  software  application  (e.g.,  software 
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program  crashes).  It  was  expected  that  the  software  failure  would  cause  the  greatest  interruption. 
“H”  provided  the  classification  for  a  hardware  error  (e.g.,  hard  drive  failure,  monitor  failure,  etc.). 


Table  12:  Breakdown  Lengths  (sec) 


Mission 

Session 

Breakdown  Type 

Start 

End 

Breakdown 
Length  (sec) 

Breakdown 
Length  (min) 

1 

INTB 

U 

00:16:44:22 

00:16:46:25 

2.1 

0.0 

MB 

U 

00:06:37:20 

00:06:48:00 

10.4 

0.2 

MB 

LOE 

00:19:19:06 

00:19:44:02 

24.9 

0.4 

MD 

S 

00:03:36:03 

00:19:46:03 

970.0 

16.2 

MD 

LOE 

00:19:45:24 

00:20:13:10 

27.6 

0.5 

MD 

S 

00:04:00:24 

00:04:09:00 

8.3 

0.1 

2 

FINAL  CORD 

U 

00:03:31:25 

00:03:36:02 

4.3 

0.1 

MB 

U 

00:03:32:03 

00:03:41:25 

9.7 

0.2 

MD 

LOE 

00:04:00:24 

00:04:09:00: 

8.3 

* — 1 

O 

MD 

S 

00:13:20:02 

00:14:24:11 

64.3 

1.1 

MIDCORD 

LOE 

00:02:31:12 

00:02:34:00 

2.6 

0.0 

MIDCORD 

U 

00:07:49:06 

00:07:53:12 

4.2 

0.1 

STMD 

LOE 

00:06:57:29 

00:08:36:03 

98.2 

1.6 

3 

MB 

U 

00:09:49:10 

00:10:10:19 

21.3 

0.4 

Average  Breakdown  Length 

89.73 

1.5 

3.4  Many-to-Many  Communication  Measures 

3.4.1  CSCW  Tools 

The  tools  available  for  use  during  the  briefing  and  debriefing  sessions  were  as  follows: 

■  Computer  operator  station  or  other  computer  workstations  (C) 

■  Polycom  system  (P) 

■  SMARTboard  (SB) 

■  Whiteboard 

■  Printer  (PR) 

■  Shredder 

■  telephones 

The  “C”  was  used  to  identify  any  computer  use  (operator  or  alternate  workstation)  in  both  SERF 
mission  session  locations.  This  included  simple  actions  such  as  advancing  the  presentation  slide. 
“P”  refers  to  the  Polycom  video  conference  system  and  “SB”  identifies  each  instance  a  participant 
used  a  SMART  board  pen  to  highlight  or  indicate  (i.e.,  write  words,  lines,  symbols,  etc.) 
information  displayed  on  the  SMART  board.  “PR”  was  used  to  identify  a  printer.  The  frequency  of 
tool  use  (i.e.,  number  of  times  each  tool  used)  is  shown  in  Table  13. 
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Table  13:  Frequency  of  Tool  Use 


Mission 

Session 

P 

SB 

C 

PR 

Total 

1 

INTB 

6 

0 

57 

0 

63 

MB 

8 

3 

90 

0 

101 

MD 

8 

1 

57 

0 

66 

Ml  Total 

22 

4 

204 

0 

230 

2 

MIDCORD 

9 

17 

24 

1 

51 

FINALCORD 

8 

11 

21 

0 

40 

MB 

9 

7 

111 

0 

127 

STMD 

9 

1 

19 

0 

29 

MD 

9 

0 

64 

0 

73 

M2  Total 

44 

36 

239 

1 

320 

3 

MB 

9 

17 

24 

0 

50 

M3  Total 

9 

17 

24 

0 

50 

Total 

75 

57 

467 

1 

600 

The  frequency  of  each  tool  was  recorded  by  counting  the  number  of  times  each  participant  used  a 
specific  tool  during  each  mission  session.  The  desktop  computer  was  the  most  frequently  used  tool 
with  a  frequency  of  467.  The  high  frequency  score  of  the  desktop  computer  is  a  direct  result  of  the 
Microsoft  PowerPoint  operator  advancing  each  slide  or  facilitating  playback  commands  for  the 
DCS  mission  log  viewer  of  Falcon  View  applications.  Even  though  the  Polycom  video  conference 
system  was  the  primary  mode  of  communication  between  the  distributed  teams,  it  only  had  a 
frequency  score  of  75.  A  count  of  “1”  was  tallied  for  each  participant  (passive  or  active)  per 
mission  session.  The  SMART  board  tool  use  increased  from  Mission  1  to  Mission  2  and  3. 

3.4.2  Turns 

The  number  of  turns  per  participant  is  reflected  in  Table  14  below.  Mission  2  contained  the  largest 
number  of  turns  compared  to  Mission  1  and  Mission  3.  The  main  reason  for  the  difference  in  turn 
count  is  that  more  sessions  were  captured  during  Mission  2  than  Mission  1  and  3  combined. 
Mission  3  had  the  lowest  number  of  turns  because  only  the  Mass  Brief  was  recorded.  PI  had  the 
greatest  number  of  turns. 

The  minimum  and  maximum,  and  quartile  spits  for  overall  turn  length  are  presented  in  Table  15. 
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Table  14:  Number  of  Turns  per  Participant 


Participant 

Mission 

Average 

Total 

1 

2 

3 

PI 

68 

306 

46 

140.00 

420 

P10 

1 

10 

4 

5.00 

15 

Pll 

7 

20 

1 

9.33 

28 

P2 

39 

CXD 
? — 1 
* — 1 

13 

56.67 

170 

P3 

199 

59 

9 

89.00 

267 

P4 

4 

3 

1 

2.67 

8 

P5 

118 

55 

14 

62.33 

187 

P6 

71 

86 

6 

54.33 

163 

P7 

62 

19 

40.50 

81 

P8 

4 

31 

2 

12.33 

37 

PW 

0 

0 

0 

0 

0 

Average  Turn  Length 

57.30 

70.70 

10.67 

Total 

573 

707 

96 

1376 

Table  15:  Turn  Length  Percentile 


Percentile 

Turn  length  (sec) 

Minimum 

0.03 

25th  percentile  (i.e.  short) 

0.76 

50th  percentile  (i.e.  medium) 

1.79 

75th  percentile  (i.e.  long) 

5.56 

Maximum 

180.03 

3.4.3  Active  Participants 

Active  participants  were  defined  as  individuals  that  were  present  during  the  mission  session 
and  contributed  verbally  to  the  information  presented  to  the  distributed  team  members  (see 
Table  16  for  active  participants  per  session).  Technology  operators  (power  point  slide 
operators)  were  not  considered  an  active  participant  unless  they  met  the  previously  stated 
requirement. 
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Table  16:  Total  number  of  active  participants  per  Mission 


Part. 

Mission  1 

Ml 

Mission  2 

M2 

M3 

M3 

Total 

INTB 

MB 

MD 

total 

MCB 

FCB 

MB 

STMD 

MD 

total 

MB 

total 

PI 

1 

1 

1 

3 

1 

1 

1 

1 

1 

5 

1 

1 

9 

P2 

1 

1 

1 

3 

1 

1 

1 

1 

1 

5 

1 

1 

9 

P3 

1 

1 

1 

3 

1 

1 

1 

1 

1 

5 

1 

1 

9 

P4 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

0 

0 

1 

P5 

1 

1 

1 

3 

1 

1 

1 

0 

1 

4 

1 

1 

8 

P6 

1 

1 

1 

3 

1 

1 

1 

1 

1 

5 

1 

1 

9 

P8 

0 

0 

1 

1 

1 

1 

1 

1 

1 

5 

1 

1 

7 

Pll 

0 

0 

1 

1 

0 

0 

0 

1 

0 

2 

1 

1 

4 

PW 

0 

0 

0 

0 

0 

0 

0 

Total 

5 

5 

7 

17 

7 

6 

6 

6 

7 

32 

7 

7 

56 

3.4.4  Participant  Transitions 

The  number  of  total  transitions  per  participant  are  represented  in  Table  17.  The  slide  transition 
count  includes  the  presenter  requests  to  the  computer  operator  to  advance  the  power  point  slide. 
Participants  1  and  2  have  a  significantly  greater  number  of  transitions  compared  to  the  other 
participants.  An  examination  of  the  detailed  phase  data  on  individual  sheets  in  the  data  file  CSCW 
time  data.xls  and  Table  17  indicates  that  those  participants  with  a  greater  number  of  transitions 
directly  relates  to  number  of  phases  a  participant  was  responsible  for  presenting  and/or  if  the 
participant  was  the  computer  operator  (i.e.,  advancing  the  power  point  presentation  slides). 
Participants  with  a  lower  number  transitions  had  fewer  presentation  responsibilities  and  were  not 
used  (or  used  rarely)  as  computer  operators.  PI  acted  as  the  briefing  and  debriefing  moderator  for 
the  majority  of  the  missions,  so  it  was  not  unexpected  to  discover  that  PI  had  the  highest  number  of 
questions  compared  to  the  other  participants. 


Table  17:  Number  of  Transitions 


Transition 

Participant 

Total 

PI 

P2** 

P3 

P5 

P6 

P7 

P8 

P11* 

PW* 

Introductions 

33 

5 

14 

52 

Questions 

96 

27 

32 

20 

30 

13 

3 

3 

224 

Slide 

156 

235 

40 

50 

7 

5 

1 

1 

7 

502 

Thank  you 

2 

4 

6 

Total 

288 

266 

90 

70 

37 

18 

4 

4 

7 

785 

*  indicates  participant  assuming  role  as  computer  operator 


3.4.5  Participant  Turn  Overlaps 

For  this  observational  study,  an  overlap  refers  to  the  instance  when  one  participant  speaks  over 
another  participant  who  is  still  talking. 
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Table  18  shows  the  number  of  overlaps  committed  by  each  participant  by  every  other  participant. 
The  “?”  indicates  instances  of  overlap  when  the  speaker  could  not  be  ascertained.  PA11  represents 
all  the  participants  laughing  in  response  to  behaviour  by  the  speaker. 

PI  was  overlapped  by  other  participants  the  most,  and  interestingly,  also  committed  the  most 
overlaps  against  others.  Which  participants  interrupt  others  and  which  participants  are  interrupted 
by  others  does  not  appear  to  covary  with  rank.  Rather,  an  inspection  of  Table  14  reveals  that  those 
participants  with  greater  number  of  speaking  turns  are  also  those  participants  who  are  most  often 
interrupted  by  others  and  those  who  interrupt  others.  Similarly,  those  participants  with  the  lowest 
number  of  speaking  turns  less  often  interrupted  and  did  not  interrupt  others. 
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Participant 

Being 

Overlapped 

PI 

PI 

*P2 

14 

P8 

2 

Pll 

4 

P4 

*P3 

10 

*P5  n 

3 

*P6 

11 

P7 

5 

PAII 

2 

Total 

51 

Lsgsesgl 


HUMAN  SYSTEMS 


Table  18:  Number  of  Overlaps 

Participant  Overlapped  Bv 


PI  PAII  Pll  *P2  *P3 


*P5  *P6 


P8  ?  Total 


2  1 


5  2 


*  indicates  participant  is  a  retired  member  of  the  CF 
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The  participants  worked  to  mutually  benefit  one  another,  by  operating  in  a  cooperative  manner  by 
sharing  and  dividing  tasks.  During  the  briefing  and  debriefing  phases,  individuals  were  working 
towards  the  same  goal,  but  it  is  unclear  if  this  is  considered  true  collaboration.  The  briefing  and 
debriefing  sessions  had  a  fairly  consistent  and  rigid  structure,  and  were  a  forum  to  present  and 
review  information  in  a  structured  process.  Some  extra  footage  that  was  captured  during  the  video 
recording  process  involved  a  small  part  of  a  planning  session.  This  video  data  was  not  coded,  but 
was  superficially  assessed  to  note  any  difference  between  the  structured  session  and  an 
unstructured  one.  The  planning  session  appeared  to  demonstrate  a  higher  level  of  collaboration 
than  the  brief  and  debrief  sessions.  Participants  were  verbally  transferring  information  to  each  other 
using  the  video  conference  system  that  demonstrated  a  higher  mutual  dependency  than  what  was 
demonstrated  in  the  brief  and  debrief  sessions.  The  CSCW  system  provided  necessary  tools  for  the 
individuals  in  different  locations  with  different  responsibilities  to  interact  and  conduct  work. 

The  orientation  of  the  SMART  board  was  used  in  coordination  and  floor  control.  All  participants  in 
the  downstairs  SERF  room  focused  their  attention  to  the  SMART  board  when  information  was 
being  presented  or  discussed.  Participants  in  the  upstairs  SERF  room  had  the  SMART  board 
information  being  displayed  on  a  computer  screen  which  did  not  command  that  same  amount  of 
attention  as  the  SMART  board.  Upstairs  participants’  attention  sometimes  was  directed  away  from 
the  presentation  and  in  some  cases,  presenters  had  to  be  prompted  multiple  times  to  address  their 
phase  during  the  brief  or  debrief.  The  SMART  board  is  an  important  CSCW  tool  because  it  has  the 
potential  to  serve  multiple  functions  other  than  focusing  attention.  The  SMART  board  can  be  used 
to  store  information,  express  ideas,  and  mediate  the  interaction  between  geographically  separated 
team  members. 

Flow  ever,  the  SMART  board  fails  to  convey  specific  and  precise  information  of  hand  gestures. 
Even  in  conjunction  with  the  Polycom  system,  hand  gestures  were  not  communicated  to 
participants  outside  of  the  location  they  were  made.  It  is  difficult  to  ascertain  why,  but  very  few 
gestures  were  made  at  all  during  the  3  missions.  All  the  participants  relied  on  the  use  of  verbal  cues 
and  the  SMART  pen  when  discussing  specific  information  displayed  on  the  SMART  board. 
Upstairs  participants  could  have  seen  gestures  made  by  downstairs  participants  (not  the  presenter) 
via  the  Polycom  video  conference  system.  However,  downstairs  participants  would  have  had 
difficulty  observing  any  gestures  made  by  participants  upstairs  because  the  Polycom  camera  was 
situated  to  close  to  the  upstairs  participants.  In  most  instances,  upstairs  participants  had  to  crowd 
around  the  Polycom  system  resulting  in  a  close-up  shot  of  their  torso  displayed  in  the  field  of  view. 

Given  the  nature  of  the  federated  CSCW  system,  the  participants  were  fortunate  not  to  experience 
any  major  hardware  or  software  breakdown.  Two  factors  that  reduced  the  negative  impact  and 
number  of  breakdowns  could  be  attributed  to  learning  effects  and  the  ability  of  the  technical 
support  person  to  move  freely  between  the  downstairs  and  upstairs  SERF  locations. 

The  ability  of  the  technical  support  person  to  move  freely  between  the  two  locations  contributed  to 
the  quick  resolution  of  any  hardware  or  software  breakdown.  Had  the  two  meeting  locations  been 
geographically  separated  by  a  large  distance,  breakdowns  would  presumably  have  taken  longer  to 
resolve.  Also,  if  the  breakdowns  were  not  resolved,  they  would  certainly  have  affected  the 
participant’s  ability  to  complete  their  work  responsibilities. 

Another  factor  in  the  small  amount  of  breakdowns  can  be  attributed  to  learning  effects.  The  more 
frequently  participants  performed  tasks  using  the  CSCW  system  the  more  efficient  they  became  at 
resolving  user  and  lack  of  operating  experience  errors.  Participants  became  more  willing  and  better 
equipped  to  assist  one  another  in  using  the  CSCW  tools  that  they  were  previously  unfamiliar  with. 
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An  example  of  this  behaviour  clearly  illustrates  this  point.  An  upstairs  SERF  room  participant  was 
having  difficulty  understanding  what  information  was  being  emphasised  on  a  map  display.  The 
participant  presenting  the  data  (downstairs  SERF  room)  was  directing  the  computer  operator  to  use 
the  mouse  cursor  to  highlight  specific  flight  path  information  on  the  SMART  board.  The  upstairs 
participant  requested  the  presenter  to  use  the  SMART  pen  on  the  interactive  whiteboard  to  make 
the  highlighting  more  obvious.  The  presenter  did  not  know  how  to  use  the  SMART  pen  effectively 
and  was  coached  by  other  briefing  participants  to  use  the  tool  correctly.  The  SMART  pens  were 
used  in  a  more  effective  manner  after  this  experience. 

The  breakdowns  that  did  occur  should  be  considered  minor  and  not  surprising  given  the  different 
hardware  and  software  components  being  integrated  together.  From  the  data  collected,  it  was 
difficult  to  assess  what  impact  the  breakdowns  (that  did  occur)  had  on  group  efficiency  and 
motivation.  Participants  appeared  willing  to  work  through  the  user  error  and  lack  of  operating 
experience  breakdowns,  but  exhibited  some  signs  of  impatience  when  the  DCS  mission  log  viewer 
did  not  function  as  expected.  A  work  around  solution  resolved  the  software  error  (run  DCS  through 
internal  SERF  networked  computers),  but  we  suspect  if  the  system  resulted  in  a  major  hardware 
breakdown,  group  efficiency  and  motivation  would  have  significantly  decreased. 
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To  accurately  assess  user  requirements  for  the  DRDC  Toronto’s  CSCW  system(s),  the  following 
areas  should  be  specifically  addressed:  simulated  scenarios  designed  specifically  to  address  CSCW 
tool  use;  group  dynamics;  task-oriented  communication;  better  integration  of  federated  tools;  and 
geographically  separated  teams. 

It  may  be  beneficial  to  construct  and  test  a  specific  hypothesis  related  to  how  CSCW  tools  affect 
different  areas  of  communication,  preparation,  and  information  sharing  within  a  military  context. 
Simulated  scenarios  could  be  created  to  identify  specific  work  tasks  and  define  what  users  are 
expected  to  do.  The  scenarios  or  work  environments  could  be  configured  so  each  environment  is 
supported  by  different  collaboration  tools  (e.g.,  use  of  interactive  whiteboard,  use  of  Polycom 
system,  audio,  no  audio,  text  messaging,  etc.).  Setup  the  scenarios  so  the  level  of  mutually 
dependent  work  varies  from  task  to  task. 

Group  dynamics  need  to  be  assessed  more  thoroughly  in  determining  user  requirements.  The 
interaction  between  strangers  versus  people  that  know  each  other  well  may  be  significantly 
different.  Within  Maple  Skies,  all  the  participants  had  some  level  of  familiarity  with  each  other 
prior  to  the  study.  The  result  of  CSCW  evaluation  may  have  differed  if  another  team  was  used  that 
was  not  familiar  with  one  another.  Participants  in  CSCW  observational  study  seemed  comfortable 
and  casual  around  one  another  leading  to  a  very  relaxed  and  comfortable  environment.  It  would  be 
worthwhile  to  evaluate  a  group  of  strangers  to  determine  if  performance  measures  are  comparable. 
Specific  user  requirements  for  the  CSCW  system  will  need  to  consider  the  familiarity  level  of 
teams. 

Evaluating  the  mission  briefing  and  debriefing  stages  will  be  useful  for  identifying  CSCW  user 
requirements  that  focus  on  task-oriented  communication  between  team  members.  Another 
important  aspect  of  team  collaboration  occurs  during  the  various  planning  stages.  From  the  phase 
data  collected,  the  briefing  and  debriefing  stages  are  very  structured.  It  may  be  beneficial  to 
determine  CSCW  user  requirements  for  a  less  ordered  environment  or  tasks  that  require 
geographically  separated  team  members  to  work  more  together  on  a  task.  That  is,  team  members 
have  to  work  collectively  and  simultaneously  to  complete  the  task.  From  the  video  data, 
participants  in  Maple  Skies  completed  their  work  outside  of  the  briefings.  It  also  could  be 
beneficial  if  there  was  a  method  for  a  member  from  each  geographically  distributed  team  to  have 
ability  to  engage  in  a  side  bar  conversation,  especially  if  the  content  of  the  exchange  is  not  relevant 
to  the  group’s  work. 

The  DCS  mission  log  viewer  needs  better  integration  into  the  CSCW  system.  The  majority  of  the 
breakdown  time  comprised  technical  assistance.  The  DCS  was  shared  through  the  SERF  network 
which  may  not  have  been  available  if  the  two  groups  had  not  been  collocated.  In  most  cases  with 
the  DCS,  it  was  not  because  of  the  user’s  lack  of  operating  experience  that  contributed  to  the 
problem,  but  rather  the  DCS  had  difficultly  integrating  with  other  CMTR  software  applications 
(e.g.,  Bridge-It).  The  advantage  of  the  federated  system  allowed  the  use  of  the  mission  log  viewer, 
but  it  significantly  affected  timely  delivery  of  the  information.  Also,  use  of  the  DCS  required  the 
skills  of  a  specialized  operator.  If  no  technical  support  was  present  at  the  time  of  operation,  the 
mission  team  members  would  not  have  received  the  detail  required  to  conduct  an  effective  debrief. 

The  geographic  distance  between  the  study  participants  was  not  sufficient.  Two  rooms  within  the 
SERF  were  used  -  a  downstairs  conference  room  and  an  upstairs  office.  The  physical  distance 
separating  the  rooms  was  easily  overcome  by  the  study  participants,  who  frequently  travelled 
between  them  during  differently  stages  of  the  mission.  To  effectively  study  and  define  user 
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requirements  for  the  associated  briefing  and  debriefing  activities,  the  CSCW  system  should  also  be 
evaluated  using  two  geographically  separated  locations.  Participants  in  this  study  experienced  a 
level  of  interaction  and  collaboration  that  would  not  be  possible  if  the  individuals  were  prevented 
from  direct  contact  with  one  another. 
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Annex  A  Questionnaires 
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Computer  Experience  Questionnaire 


HUMANSlSr£.l/S 


Please  answer  the  following  questions  by  placing  an  X  in  the  location  on  the  line  that 
corresponds  to  your  computer  experience. 

1.  What  is  your  frequency  of  computer  usage? 


Never  Daily 


2.  What  is  your  frequency  of  Office  suite  software  usage?  (e.g.  Word,  PowerPoint,  Excel) 


Never  Daily 


3.  How  comfortable  are  you  with  Office  suite  software?  (e.g.  Word,  PowerPoint,  Excel) 


Uncomfortable 
(I  don ’t  know  how 
to  do  anything) 


Very  Comfortable 
(I  know  shortcuts 

for  many  things) 
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Computer-Supported,  Collaborative  Work  Survey 

Name: 


Day:  Wednesday 


Thursday 


Friday 


Brief  or  Debrief: 


Briefing 


Debriefing 


Based  on  the  (de)briefwg  that  yon  just  attended,  please  mark  your  responses  to  the  questions  below. 
CB1:  Was  the  main  goal  of  the  briefing  accomplished? 

YES  NO 


If  NO,  what  was  the  major  factor  prohibiting  the 
completion  of  the  goal? 


CB2:  Could  you  identify  all  of  the  aircrew,  at  local  and  remote  sites,  involved  in  the  briefing? 
YES  NO 

If  YES,  how  many  people  were 
at  the  remote  location(s)? 


CB3:  Could  you  see/follow  the  material  (documents,  presentations,  etc)  that  were  being  referenced  by  the 
presenter? 

YES  NO 

If  NO,  what  was  the  major  difficulty  in  following  the  presenter  and 
reference  material? 


CB4:  Could  you  tell  if  remote  people  could  be  interrupted? 

YES  NO 

If  NO,  what  was  the  major  difficulty  in  obtaining  this 
information? 


CB5:  Could  you  tell  when  other  people  were  ready  to  continue  to  the  next  topic  of  the  briefing  (e.g.,  after 
detailed  flight  plans  were  uploaded)? 

YES  NO 

If  YES,  how  did  you  obtain  this  information? 


CB6:  Would  private  communication  with  a  participant  from  the  other  remote  locations(s)  be  of  benefit? 
YES  NO 

If  NO,  could  you  envision  a  scenario  would  be  of  benefit 
during  a  distributed  briefing? 
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Generic  Measures  of  Performance  -  Mass  Brief  and  Mass  Debrief 


HUMANS  KSTf  MS 

Imtorpors  Ii4 


For  each  measure  listed  below,  please  circle  the  category  (i.e.  Poor,  Marginal,  Standard, 
Very  Good,  and  Exceptional)  that  maps  to  the  actions  you  just  observed.  Use  the 
behavioural  descriptions,  and  “Look  For’s”  to  direct  your  choice.  If  you  feel  the  measure  is 
Not  Applicable,  then  write  N/A  in  a  blank  space  somewhere  on  the  measure.  Please 
place  a  response  for  every  measure. 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A1.  Tactics:  Task  and  ROE  understanding,  route  review/analysis 

Accepted  mission 
tasking  without 
review  of  important 
information 

Accepted  mission 
tasking  with  some 
review  of  key 
information 

Good  review  of 
important 
information  and 
understood  the 
mission  tasking 

Carefully  analyzed 
the  mission  tasking 
and  important 
information 

Completely 
analyzed  the 
mission  tasking 
and  all  related 
information 

Logical  deductions 
and  inferences 
were  made  based 
on  available 
information 

Look  for: 

Thorough  review  of  Air  Tasking  Order  (ATO),  Airspace  Coordination  Order  (ACO),  Intelligence  update 

Understood  where  mission  fits  in  overall  big-picture  plan 

Thorough  review  of  target  area  and  routing 

Thorough  review  of  ROE,  FLIP,  NOTAMS 

Threat  analysis/update 

Aware  of  significant  terrain  or  hazards  to  flight  along  planned  routing 

Ingress/egress  corridors,  location  of  friendly  forces. 

Ensured  all  remote-site  participants  included  where  appropriate? 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A2.  Tactics:  Factors  considered  in  plan,  development  of  mission  tools. 

No  consideration  of 
key  planning 
factors 

No  additional  tools 
added  to  mission 
execution  plan 

Some  discussion 
of  planning  factors 

Minimal  tools 
added  to  mission 
execution  plan 

Identified  and 
analyzed  many  key 
planning  factors 

Incorporated  some 
tools  into  mission 
execution  plan 

Identified  and 
analyzed  all  key 
planning  factors 

Integrated  many 
additional  tools  into 
the  mission 
execution  plan 

Challenged/ 
questioned  plan 
assumptions 

Provided  a  detailed 
analysis  of  all  key 
planning  factors 

All  tools  were 
incorporated  into 
the  mission 
execution  plan 

Look  for: 

Were  key  planning  factors  or  inputs  omitted,  if  so  for  what  reason? 

Was  there  due  consideration  of  all  factors  in  tactical  plan  selection? 

Was  the  plan  selection  discussed  with  other  crewmembers/  wingmen  and  were  valuable  inputs  considered? 

Tactics  considered  for  each  mission  phase? 

Did  the  tactics  planned  conform  to  the  big-picture  plan. 

Were  all  potential  threats  analysed  and  proper  tactics  incorporated. 

Were  environmental  conditions  planned  for.  le  cloud  cover,  in-flight  visibility,  terrain  masking 

Ingress/egress  corridor  selection 

Wingmen  planning  responsibilities  defined 

Ensure  all  non-site  participants  included  where  appropriate? 

1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A3.  Tactics:  COA  and  contingency  plan  considers  performance  factors,  malfunctions, 
weather,  alternate  airfields 

Accepted  mission 
plan  with  no 
consideration  of 
"what-if 
contingencies 

Some  discussion 
of  "what-ifs" 

No  attempt  made 
to  incorporate 
contingency 
options  into 
mission  plan 

Identified  some 

"what-if 

contingencies 

Incorporated  these 
into  mission  plan 

Identified  many 

"what-if 

contingencies 

Incorporated  most 
into  mission  plan 

Provided  detailed 
consideration  of 
"what-if" 
contingencies 

Integrated  these 
into  mission  plan 

Look  for: 

Planned  for  what  if s  in  the  primary  mission  plan 

Planned  for  secondary  /  alternate  missions  and  applicable  what  if s 

Planned  for  battle  damage,  missing  wingmen,  late  arrivals 

Planned  departures  /  arrival  contingencies  (  ex.  Delayed  take-off, .holds  etc) 

Weather  contingencies  /  abort  plan/  alternate  airfields 

Alternate  tactics  considered  for  each  mission  phase? 

What  if’s  related  to  intelligence  brief 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A4.  Tactics:  Use  of  planning  tools:  reference  to  planning  items  in  mission  planning  kit 

Little  or  no 
reference  to 
available  planning 
tools  in 
constructing 
mission  plan 

Periodic  reference 
to  some  tools  in 
constructing 
mission  plan 

Frequent  reference 
to  some  of  the 
tools  in 
constructing 
mission  plan 

Extensive 
reference  to  most 
of  the  available 
tools  in 

constructing  the 
mission  plan 

Extensive 
reference  to  all 
available  tools  in 
constructing  the 
mission  plan 

Look  for: 

Use  of  SOPs 

Use  of  pre-planned  material  /  tools 

Use  and  adherence  to  current  tactics  manual  /  Airplane  Operating  Manual 
Use  of  applicable  computer  programs  for  mission  planning 
NOTAMs,  hazards,  and  in-flight  weather  updated 
Check  primary  airfields  and  alternate  for  suitability 

Check  maps,  target  area  photos,  satellite  imagery  and  FUR  pics  for  accuracy 

Request  specific  resources  required  for  planning,  ie  satellite  imagery,  reconnaissance  photos  etc. 

Was  the  material  visible/accessible  to  both  local  and  remote  sites? 

Did  the  presenter  make  effective  use  of  the  tools? 

Were  there  appropriate  communications  with  remote  mission  participants  prior  to  briefing? 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A5.  Time  management:  time  appreciation,  efficiency  in  time  spent  planning  to 
accomplish  all  required  planning  activities 

No  time 
appreciation 
carried  out 

Spent  minimal  time 
planning 

Much  of  the  time 
was  unproductive 

Cursory  time 
appreciation 
carried  out 

Spent  more  than 
the  minimal  time 
planning  but  not 
much 

Considerable 
unproductive  time 

Time  appreciation 
was  carried  out 

Spent  adequate 
time  planning 

Does  not  use  all 
available  time 

Good  time 
appreciation 
considering  most 
inputs 

Spent  considerable 
time  planning 

Made  effort  not  to 
have  unproductive 
time  during  the 
planning  session 

Ideal  time 
appreciation  made 
(including  due 
consideration  for 
non-standard 
timings  and  inputs) 

Used  all  available 
time  planning 

Little  or  no 
unproductive  time 
during  the  planning 
session 

Look  for: 

Adequate  time  appreciation  made  with  due  consideration  for  non-standard  inputs 

Time  appreciation  and  plan  communicated  to  other  formation  members 

Time  appreciation  monitored  and  revised  during  planning  process 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A6.  Function  allocation  and  Team  Performance 

Flight  lead 
provided  no 
direction  for 
planning 
responsibilities 

Wingman  engaged 
in  planning  with  no 
discussion  of  roles 

Flight  Lead  gave 
limited  direction  for 
planning 

responsibilities  and 
expectations 

Flight  lead  gave 
good  direction  to 
all  flight  members 
concerning 
planning 
responsibilities 

Flight  lead 
provided  clear 
direction  to  all  flight 
members 
concerning 
planning 

responsibilities  and 
guidance  on  time 
to  be  spent  in 
planning 

Flight  lead  outlined 
expectations  for 
planning  products 

Flight  lead 
reviewed  crew 
products 

Flight  lead 
provided  very  clear 
direction  to  all  flight 
members  for 
planning 

responsibilities  and 
guidance  on  time 
to  be  spent  in 
planning 

Flight  lead 
monitored  planning 
process  of 
wingmen  and 
provided  timely 
and  appropriate 
feedback 

Flight  lead 
reviewed  all 
wingmen  products 
and  all  errors  were 
rectified 

Look  for: 

Responsibilities  were  delegated  by  the  mission  lead  iaw  SOPs 

Responsibilities  were  delegated  according  to  individual  strengths 

Expectations  of  Flight  lead  were  clear  and  completion  timings  given. 

Wingment  planning  activities  were  appropriately  supervised 

Planning  products  were  well  reviewed,  errors  were  found,  explained  and  corrected 

Flight  lead  able  to  supervise  wingmen  without  sacrificing  own  responsibilities 

1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A7.  Quality  of  planning  products:  Fuel  plan:  level  of  detail,  quality  of  plan,  use  of 
computer  products 

Accepted  fuel  plan 
as  given  with  no 
further  observation 

Checked  fuel  plan 

Made  no  additional 
adjustments 

Checked  fuel  plan 

Considered  making 
some  adjustments 
based  on 

additional  planning 
factors  (required 
alternate,  weather 
changes  etc) 

Considered  fuel 
plan  against  a 
range  of  options 
(e.g.,  engine  loss, 
unexpended 
ordnance,  low-level 
ops  etc.) 

Included  all 
mission  critical  and 
environmental 
variables  in  fuel 
plan 

Look  for: 

Considered  primary  plan  and  contingencies  (what  if  no  air-to-air  refuelling,  weather  enroute,  engine  failure 
enroute).  Ensured  fuel  available  is  enough  to  conduct  mission  safely. 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A8.  Quality  of  planning  products:  TOLD:  consideration  of  airfield  info,  obstacles, 
performance  factors,  NOTAMs,  fuel  load 

Computed  TOLD 
with  major  errors 
and  omissions 

Computed  TOLD 
with  several  errors 
and/or  omissions 

Computed  TOLD 
with  minimum 

errors 

Minimum  mission 

requirements 

addressed 

Computed  TOLD 
with  no  errors 

Most  mission 

variables 

addressed 

Computed  TOLD 
with  no  errors 

All  mission 

variables 

addressed 

Look  for: 

Computed  TOLD  parameters  for  departure  and  arrival  airfields  (including  alternates). 

Included  other  flight  members  in  TOLD  factor  calculation 

Is  TOLD  for  future  events  based  on  future  meteorological  data,  or  planned  using  current  data? 

Updates  in  weight,  weather,  runway  surface  conditions,  barometric  pressure  and  temperature  considered? 
Is  TOLD  calculated  for  contingencies  (backup  aircraft  configurations,  fuel  loads,  weapons  loads) 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A9.  Quality  of  planning  products:  Comm  plan:  level  of  detail,  quality  of  plan,  use  of 
computer  products 

Accepted  comm 
plan  as  given  with 
no  further 
consideration 

Checked  comm 
plan 

Made  no  additional 
adjustments 

Checked  comm 
plan 

Considered  making 
some  adjustments 
based  on 

additional  planning 
factors  (IFR/VFR, 
INTER-FLIGHT 

Considered  comm 
plan  against  a  wide 
range  of  options 
(e.g.,  AWACS, 

GCI,  Comm- 
jamming  etc. 

Included  all 
mission  critical  and 
environmental 
variables  in  comm 
plan 

Look  for: 

Understands  complex  C2  structure 

Indicated  mission  critical  comms 

Considerations  for  alternate  comm  plan,  secure  requirements,  monitoring  of  common  /  important 
frequencies,  comm  jamming,  comm  failure 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

A10.  COMMUNICATION:  Mission  briefings:  detail,  comprehensiveness,  overall 
effectiveness 

Little  or  no  mission 
briefing. 

Time  wasted  on 

irrelevant 

communication 

Overall  plan  not 
explained 

Minimal  mission 
briefing. 

Most  elements 
covered. 

Provided  little 
detail  on  objectives 

Overall  plan  not 
well  explained 

Brief  contained 
some  inconsistent 
mission  details 

Overall  plan 
explained  with 
some  detail 

Detailed  briefing 

Many  elements 
covered  with 
acceptable  level  of 
detail 

Overall  plan  well 
explained  and 
understood 

Highly  detailed 
mission  brief. 

All  mission 
elements  covered 
with  great  detail 

Overall  plan  very 
well  explained, 
communicated  and 
understood  by  all 
flight  members 

Look  for: 

Were  the  mission  objectives  clearly  stated  and  accurate. 

Was  key  info  omitted? 

Were  there  parts  of  the  brief  requiring  more  detail? 

Were  references  to  SOPs  used  to  avoid  “over-briefing” 

Was  the  time  available  well  used  to  cover  most  critical  items  of  the  mission  and  were  they  well  prioritized? 
Was  it  rushed? 

Was  it  well  structured  or  fragmented  and  un-organized? 

Was  the  actual  tactical  plan  well  explained? 

Do  flight  members  appear  to  leave  briefing  with  understanding  of  mission?  -  with  confidence? 

Were  there  appropriate  communications  with  remote  mission  participants? 

Proportion  of  time  spent  on  domestics*  and  time  spent  on  actual  tactical  plan 
*  domestics  -  includes  from  take-off  to  tactical  area  and  tactical  area  back  to  landing 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

All.  Technology  mediated  communication:  consider  the  effectiveness  of  devices 
such  as  Smartboards  and  video-conferencing,  during  distributed  meetings. 

Time  was  spent 
addressing 
technical  problems 
negatively  affected 
understanding 
mission  plan  or 
lessons  learned. 

Remote 

participants  did  not 
receive  the  same 
information  as  local 
participants. 

Interaction  with 
devices  distracted 
from  the  goal  of  the 
meeting. 

Some  time  was 
spent  addressing 
technical 
difficulties. 

Had  to  wait  for 
remote  participants 
to  receive 
information. 

Poor  transitions 
between  briefings 
due  to  technical 
difficulties. 

Some  time  was 
spent  addressing 
technical 
difficulties. 

Local  and  remote 
participants 
received  the  same 
information. 

Incorporates  some 
devices  into 
mission  execution 
plan. 

The  use  of  the 
devices  did  not 
distract  from  the 
meeting. 

Similar  experience 
as  an  “in  person” 
meeting. 

Integrates  many 
additional  devices 
into  the  mission 
execution  plan. 

Remote 

participants  had 
same  level  of 
input/interaction  as 
local  participants. 

Devices  were 
seamless  used  in 
the  presentation. 

Devices  used  in 
the  meeting  was 
optimal  for  the 
presentation  of  the 
mission  execution 
plan 

Look  for: 

Could  you  identify  all  air  crew  participating  at  the  remote  sites? 

Could  you  tell  if  remote  people  could  be  interrupted? 

Could  you  tell  when  remote  participants  were  ready  to  continue  to  the  next  topic  of  the  briefing? 

Was  there  a  need  for  private  communication  with  participants  at  a  remote  site? 

Was  the  transition  between  briefings  seamlessly  accomplished? 

1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

AO.  Overall  Planning  and  Briefing 

1  Behaviour 

Planning 
behaviours  were 
significantly  below 
expectations. 

This  level  of 
proficiency  is  less 
than  desired  for 
effective 

coordination  during 
the  mission.  There 
is  room  for  much 
improvement. 

Behaviour 
promoted  and 
maintained 
coordination  and 
mission  operations 
effectiveness.  This 
is  the  minimum 
standard  level  of 
proficiency  that 
should  be 
expected  during 
this  mission. 

Planning 
behaviours  were 
significantly  above 
expectations. 

Behaviours 
represent  a  high 
level  of  skill  in  the 
application  of 
specific 

behaviours,  and 
serves  as  a  model 
for  coordination, 
teamwork  and 
highly  efficient 
mission  operations. 
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Mission  Execution 

Generic  Measures  of  Performance 

For  each  measure  listed  below,  please  circle  the  category  (i.e.  Poor,  Marginal,  Standard, 
Very  Good,  and  Exceptional)  that  maps  to  the  actions  you  just  observed.  Use  the 
behavioural  descriptions,  and  “Look  For’s”  to  direct  your  choice.  If  you  feel  the  measure  is 
Not  Applicable,  then  write  N/A  in  a  blank  space  somewhere  on  the  measure.  Please 
place  a  response  for  every  measure. 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B1.  Plan  compliance:  Navigation  accuracy:  awareness  of  current  location,  adherence 
to  plan  considered 

Pilot  was 
lost/disoriented 

No  adherence  to 
planning  routing 

Unable  to  meet 

objective 

requirements 

Often  deviated 
from  /  is  unsure  of 
routing 

Was  only  able  to 
meet  objective 
requirements  with 
difficulty 

Generally  able  to 
follow  the  planned 
routing 

Several  off-track 
deviations 
performed  to  meet 
objectives 

Was  able  to 
adhere  to  planned 
mission  routing 

Was  aware  of 
position  with 
respect  to 
objectives  at  all 
times 

Was  continually 
aware  of  position 
and  anticipating 
routing  ahead 

Able  to  make 
timely  and 
appropriate 
adjustments  to 
meet  objective 
requirements 

Look  for: 

Accurate  position  with  reference  to  planed  route.  To  include  mission  target 

Accurate  position  (with  reference  to  overall  mission  eg.  Other  aircraft) 

Appropriate  routing  corrections  adjustments  made  (necessary  for  mission  completion) 

Flight  members  adapt  to  the  current  situation 

Crew  anticipation  of  routing  ahead 

Perform  adequate  checks  on  nav  data  entered  into  flight  computer 

Any  questioning  of  flight-members  with  regard  to  navigational  position 

1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B2.  Plan  compliance:  Time  control:  planned  timings  were  met 

Missed  many 
important  time 
milestones 

Was  continually 
"behind  the 
mission" 
throughout 

Missed  several  key 
time  milestones 

Hit  all  major  time 
milestones 

Missed  several 
minor  ones 

Hit  all  time 
milestones 

Hit  all  time 
milestones  and 
was  continually 
"ahead  of  the 
mission" 
throughout 

Look  for: 

Mission  starts  on  time;  on-station  on  time;  push  on  time 

Hit  ETAs  (eg.,  tanker  times,  time-on-target,  )  within  appropriate  time  deviation. 

Ability  to  analyse  and  correct  timing  deviations 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B3.  Communications  systems  usage:  stay  on  correct  frequency,  talk  to  proper 
agency,  correct  terminology 

Frequent  problems 
in  contacting 
proper  agency  or 
using  correct 
frequency 

Periodic  problems 
in  contacting 
proper  agency  or 
using  correct 
frequency 

No  major  problems 
in  contacting 
proper  agency  or 
using  correct 
frequency 

Proper  agencies 
contacted,  correct 
frequencies,  code 
words  and  call 
signs  were  used 
throughout 

Clear,  efficient 
communications 
executed 
throughout  entire 
mission 

Look  for: 

Correct  frequency,  correct  agency  used/contacted  throughout 
Get  all  clearances  when  needed 
Use  of  appropriate  call  signs? 

Use  of  appropriate  code  words? 

Correct  terminology? 

Clear  concise  phrasing? 

Transmit  only  when  situ  required?  Number  and  frequency  of  comms  appropriate  to  situation? 

Responds  appropriately  and  promptly  when  contacted  by  external  agency/formation  member(s) 

Flight  members  question  communication  when  not  in  agreement  with  what  is  being  observed 
Properly  monitors  all  appropriate  frequencies 
No  broadcast  or  specific  information  missed 

Followed  proper  corn-jamming  procedures  (refers  to  section  4c  in  planning) 

1.  Poor  2.  Marginal  3.  Standard  4.  Very  Good  5. Exceptional 

B4.  Aircraft  handling  and  control:  airspeed,  altitude,  heading 

Excessive  flight  Some  flight  path  Few  flight  path  Minimal  flight  path  Minimal  flight  path 

path  deviations  deviations  which  minor  deviations  deviations  which  deviations  which 

which  were  not  were  not  corrected  which  were  were  corrected  in  a  were  anticipated 

corrected  in  a  in  a  timely  manner  corrected  in  a  timely  manner  and  corrected  in  a 

timely  manner  timely  manner  timely  manner 

Look  for: 

Maintains  adequate  terrain  clearance. 

Maintains  safe  separation  from  other  aircraft. 

Maintains  correct  flight  parameters  (ground  speed,  altitude,  heading)  throughout  the  mission. 
Speed/accuracy  of  corrections  to  observed  errors  in  aircraft  flight  parameters. 

Speed/accuracy  of  changing  flight  parameters  when  requested  by  external  agencies  (ATC,  AWACS,  Flight 
Lead). 

Anticipation  of  required  changes 

Optimum  aircraft  performance  attained  and  maintained 

Degree  to  which  over-corrections  to  observed  errors  were  made 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B5.  System  awareness:  Checklist  accomplishment:  checklists  accomplished  in  a 
timely,  accurate  manner 

Fails  to  complete 
many  Emergency 
and/or  normal 
procedures/ 
checklists 

Most  Emergency 
and/or  normal 
procedures/ 
checklists 
complete 

Not  timely  or 
missed  items 

All  required 
Emergency  and/or 
normal  procedures/ 
checklists 
complete 

All  required 
Emergency  and/or 
normal  procedures/ 
checklists  were 
completed  in  timely 
manner 

Covers  all  items 

All  Emergency 
and/or  normal 
procedures/ 
checklists 
completed  in  timely 
manner  or  early 

Efficiently  covers 
all  items 

Look  for: 


Ability  to  perform  normal  procedures  items  by  memory  without  mistakes 

Completed  before  start  checklist,  before  taxi  checklist,  taxi  checklist,  before  takeoff  checklist,  arming  point 
checklist,  line  up  checklist  on  time 

Completed  all  tactical  checklists  properly  and  at  the  appropriate  time  (fence-in  check,  Go/No  Go  checklist, 
air-to-ground  checklist,  fence-out  checklist) 

Complete  descent  checklist,  before  landing  checklist,  after  landing  checklist,  and  de-arming  checklist  on 
time 

Completed  all  necessary  emergency  checklists  (if  required)  in  a  timely,  safe,  and  efficient  manner 
Completed  all  emergency  memory  /  recall  items  timely  and  accurately 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B6.  System  awareness:  sensors,  status  indicators 

Failed  to  recognise 
critical  changes  in 
sensors  or  status 
indicators 

Slow  to  recognise 
critical  changes  in 
sensors  or  status 
indicators 

Recognised  and 
responded  to 
critical  changes  in 
status  or  sensors 

Integrated  and 
correlated 
information 
between  sensors 

Used  available 
information  quickly 
and  efficiently 
according  to  an 
expert  strategy 

Look  for: 

Awareness  of  overall  aircraft  status  and  indications  including  all  operational,  degraded,  and  unserviceable 
items  (fuel,  flight  controls,  avionics,  aircraft  subsystems,  weapons) 

Awareness  of  sensor  status  and  information  (radar,  FUR,  RWR,  data  link,  jammers,  IFF  interrogator,  etc). 
Awareness  of  what  impact  specific  changes  to  aircraft  or  sensor  status  will  have  on  overall  mission. 

Amount  of  time  required  to  notice  changes  to  aircraft  system/sensor  status. 

Ability  to  correlate  information  between  different  sensors  /  indicators 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B7.  Resource  and  flight  member  awareness 

Fails  to  detect 
changes  in 
wingmen  status  or 
mission  resources 

Missed  some 
changes  in 
wingmen  status  or 
mission  resources 

Detects  critical 
changes  in 
wingmen  status  or 
mission  resources 

Rapidly  detects 
critical  changes  in 
wingmen  status  or 
mission  resources 

Responds  to 
wingmen 
communications 
regarding  status 

Planned/ 
compensated  for 
changes  in  crew 
status  or  mission 

resources 

Proactively  scans 
for  info  regarding 
changes  in  crew 
status  or  mission 

resources 

Clearly 
sets/reminds 
wingmen  roles 

Look  for: 

Ability  to  detect  formation  members  divergence  from  mission  plan 

Formation  integrity  monitored  and  maintained 

Awareness  of  position  and  actions  of  other  formation  members  throughout  the  mission 

Awareness  of  status  of  critical  mission  resources  and  actions  taken  when  mission  resources  change 
Communications  re  flight  member  status  missed 

(For  Mission  Commander/OC)  maintaining  awareness  of  other  formations  in  the  overall  plan 

Accurate  and  timely  directions  given  to  flight  members  /  mission  resources  with  changes  in  tactical  situation 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B8.  Tactical  awareness:  contact  detection,  spatial  awareness,  co-ordination, 
monitors  formation,  applies  other  COA 


Fails  to  detect 

Frequently  out  of 

Detected  at 

Detected  enemy  at 

Flied  optimum 

contact  in  timely 

position 

appropriate  range 

earliest  opportunity 

engagement  profile 

manner 

Employs  incorrect 

Recognised  enemy 

Monitored 

Co-ordinated 

Lost  sight  of 

tactics 

tactics 

formation 

effectively  with 

friendly/enemy 

entities 

Engagement 
quality  dropped 
during  comms 

Could  not  asses 
and  fly  at  same 
time 

Did  not  co-ordinate 

Adopted  correct 

Adopted 

formation 

with  formation 

tactics 

appropriate  tactics 
to  changing  tactical 
situation 

Anticipated  enemy 

Look  for: 

Ability  to  detect  contacts  with  sensors 


Awareness  of  position  relative  the  mission  routing,  enemy  formations,  friendly  assets 
Ability  to  understand  local  tactical  environment  and  choose  effective  COAs 
Ability  to  monitor  and  direct  other  formation  members  COAs 
Awareness  of  enemy  formations  actions,  and  anticipation  of  outcome 
Maintaining  pre-briefed  roles  or  dictating  any  change  of  roles  as  required 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B9.  Mission  and  goal  awareness: 
responds  to  changes  in  mission  | 

re-establishes  mission  goals,  detects  and 
picture 

Failed  to  detect 
changes  in  mission 
situation  following 
engagement 

Poor  resumption  of 
nav  plan/join  of 
formation  following 
engagement 

Recognised 
change  in  mission 
picture 

Unsure  of 
appropriate  COA  in 
response  to 
change 

Unsure  of  mission 
goals  at  various 
points  in  mission 

Recognised 
changes  in  mission 
situation  and 
adjusts 
appropriately 

Followed  pre- 
briefed 

contingency  plan. 

Integrated 
information  to 
quickly  recognise 
changes  in  mission 
picture 

Rapidly  updated 
plan  and 
communicated 
changed  picture 
and  plan 

Anticipated 
changes  in  mission 
picture 

Had  contingency 
mission  goals 

Look  for: 

Maintaining  a  broad  scan  of  info  sources  and  reacted  accordingly. 

Ability  to  comprehend  current  changes  in  the  tactical  environment  and  apply  pre-briefed  contingency  plan. 

If  no  pre-briefed  contingency  plan,  able  to  formulate  and  communicate  a  COA  to  ensure  mission  success. 

Ability  to  anticipate  the  effect  of  current  changes  in  the  tactical  environment  on  future  events  in  the  mission. 

Appropriate  assessment  of  tactical  situation  and  use  of  defensive,  offensive  and  neutral  manoeuvring  tactics 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

BIO.  Engagement  skills:  aircraft  handling,  energy  management,  gained/maintained 
offensive  advantage 

Constant  and  large 
deviations  from 
ideal  energy  state 

Quickly  lost  clearly 
offensive  position 

Was  ineffective  to 
counter  bandits 
offensive 

Many  deviations 
from  ideal  energy 
state 

Maintained  position 
but  was  unable  to 
fly  to  a  weapon 
engagement  zone 
or  to  leave  the 
engagement 

Good  energy 
management  and 
A/C  manoeuvering 

Maintained 
offensive  position 
and  employed 
weapons 

Countered  bandits 
offensive 

manouvers 

Very  sound  energy 
management  and 
aircraft 

manoeuvering 

Quickly  capitalized 
on  offensive 
position 

Effectively 
countered  bandits 
offensive  to  a 
neutral  position 

Ideal  Aircraft 
energy 

management  and 
manoeuvering 

Expeditiously 
capitalized  on 
offensive  position 

Quickly  reversed 
from  a  defensive  to 
an  offensive 
position 

Look  for: 

Maintained  visual  with  formation  and  maintained  tally  on  bandit  or  bandits. 

Aircraft  airspeed,  g,  and  angle  of  attack 

Reactions  to  bandits  manoeuvering  and  energy  state 

Appropriate  game  plan  used 

Recognition  of  turning  room 

Lift  vector  placement 

Divergence  for  ideal  energy  state 

Awareness  of  altitude 

Quick  exit  after  engagement  or  when  opportunity  presented 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B11.  Weapon  Employment  Air-toAir:  recognition  of  weapons  employment 
opportunities,  satisfied  ROE,  validity  of  shots  at  trigger  squeeze 

Did  not  recognize 
entry  into  WEZs 

Failed  to  employ 
weapons  according 
to  the  briefed  plan 

Most  shots  did  not 
meet  all  shoot 
criteria  at  trigger  / 
release 

Did  not  recognize 
some  weapons 
employment 
opportunities 

Generally  did  not 
follow  weapon 
employment  plan 

Some  shots  did  not 
meet  weapons 
release  criteria 

Recognized  entry 
into  most  WEZs 

Generally 

employed  weapons 
according  to 
briefed  criteria 

Met  all  weapon 
shoot  criteria  at 
trigger  /  release 

Recognized  all 

WEZs 

Employed 
weapons  according 
to  briefed  criteria 

Met  all  weapon 
shoot  criteria  at 
trigger  /  release 

Appropriately 
assessed 
requirement  for 
follow  on  shots 

Anticipated  and 
recognized  all 

WEZs 

opportunities 

Always  employed 
optimum  weapon 
according  to 
briefed  first  /  ideal 
opportunity 

All  criteria  met  for 
all  weapon 
releases 

Always  anticipated 
and  recognized 
requirement  for 
follow-on  shots 

Recognized  all 
degraded  /  Lower 

PK  situations 

Look  for:  WEZ  (Weapon  Engagement  Zone) 

WEZ  entries  and  anticipation 

Missed  weapons  opportunities 

Proper  weapon  selected  and  employed 

Weapon  Pk  at  trigger  /  release  (%  of  RNE,  Range  No  Escape) 

All  release  criteria  met  at  trigger 

Recognition  of  follow-on  shot  requirement 

Validated  kills  based  on  kill-criteria 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B12.  Weapon  Employment  Air-to-Ground  identified  proper  target,  valid  delivery 
parameters,  released  weapons  on  target 

Did  not  identify 
proper  target 

Failed  to  meet 
delivery 
parameters  for 
successful  weapon 
delivery 

Released 
weapons  off 
target(ie,  wrong 
pickle  point) 

Identified  target  but 
not  in  position  for 
desired  weapon 
release 
parameters. 

Weapon  release 
marginally  on 
target. 

Identified  target 

Generally  met 

delivery 

parameters 

Weapon  release 
on  target 

Identified  target 
with  time  to  spare 

Most  delivery 
parameters  met 

Weapon  release 
right  on  target. 

Identified  target 
with  time  to  spare 

All  delivery 
parameters  met 

Weapon  release 
right  on  target 

Look  for: 

Late  target  identification 

Pressing  a  bad  attack 

Awareness  of  other  flight  members 

Respect  for  fragmentation  envelops 

Proper  switchology 

Target  channelization 

Human systems®  Incorporated 


CSCW  Tool  Usage 


A  -  18 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B13.  Threat  Reactions:  recognition  of  exposure  to  threats,  effectiveness  of  threat 
reactions,  knowledge  of  threat  performance/capabilities 

Failed  to  avoid  or 
recognize  most 
threats 

Did  not  perform 
appropriate 
counter 
manoeuvers  to 
threats 

Did  not  react 
appropriately  when 
enemy  employed 
weapons 

Recognized  most 
threats 

When  threat 
recognized,  did  not 
always  perform 
appropriate 
counter 

manoeuvers 

Did  not  react  to  all 
enemy  weapons 
opportunities 

Avoided  some 
threats 

Recognized  all 
threats 

Performed 

appropriate 

counter 

manoeuvers 

Reacted  to  enemy 

weapons 

opportunities 

Avoided  most 
threats 

Quickly  recognized 
all  threats 

Effectively 

performed 

appropriate 

counter 

manoeuvers 

Denied  enemy  of 
most  weapons 
opportunities  or 
quickly  reacted 
accordingly 

Avoided  all  non¬ 
necessary  threats 

Anticipated  all 
potential  threats 
and  proactively 
performed  ideal 
counter 

manoeuvers 

Denied  enemy  of 

weapons 

opportunities 

Look  for: 

Anticipation  and  recognition  of  threats 

Pre  emptive  counter  manoeuvrers  (terrain-mask,  chaff-weave  ,  pre-emptive  flares) 

Appropriate  counter  manoeuvers  performed  when  approaching  threats  (beam, drag,  terrain-mask) 

Effectiveness  of  Counter  manoeuvers 

Deny  enemy  weapons  opportunities 

Ability  to  assess  follow  on  threats  when  reacting 

Employment  of  appropriate  counter  measures  (Chaff,  Flares,  Jammers,  IRCCM) 

Maintain  formation  integrity  post  threat  reaction. 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B14.  Role  discipline:  ability  to  fulfill  assigned  role  within  the  mission,  ability  to  follow  the 
briefed  plan,  ability  to  support  other  crew/formation  members  during  mission 

Unable  to  perform 
or  was  ineffective 
at  most  individual 
responsibilities 

Unable  to  support 
others  or  negative 
contribution 

Mostly  contributed 
negatively  by 
assuming  wrong 
responsibilities 

Produced  chaos  in 
the  formation 

Missed  some 
critical  individual 
responsibilities 

Partially  supported 
others  when 
required 

Sometimes 

contributed 

negatively  by 

assuming 

unnecessary 

responsibilities 

Performed  all 

individual 

responsibilities 

Supported  others 
most  of  the  time 

Was  able  to 
assume  other  role 
responsibilities 
some  of  the  time 

Performed  all 
individual 

responsibilities  with 
high  standard 

Timely  supported 
others  when 
required 

Effectively 
assumed  other  role 
responsibilities 
when  required 

Ideally  and  timely 
performed  all 
individual 
responsibilities 

Anticipated 
degrading 
situations  or 
opportunities  and 
immediately 
supported  others 
when  required 

Instantly  and 
ideally  assumed 
other  role 
responsibilities 
when  required 

Look  for: 

Performance  of  individual  responsibilities 

Assertiveness  in  role 

Respecting  other  roles  responsibilities 

Timely  response  and  support 

Ability  to  perform  other  roles,  additional  duties 

Overall  contribution  to  mission  success 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

B15.  Achievement  of  primary  mission  objectives 

Primary  mission 
objective(s)  not 
accomplished 

Did  not  survive  the 
mission 

Some  mission 
objective(s)  not 
accomplished 

Survivability 
jeopardized  on 
several  occasions 

Mission 

objective(s) 

satisfactorily 

accomplished 

Minimal  threats  to 
survivability 

Mission  objectives 
easily 

accomplished 

Did  not  place 
survivability  of 
formation  at  risk 

All  mission 
objectives 
accomplished  in  an 
optimum,  safe, 
efficient  manner 

Ensured 

survivability  of  own 
and  other  mission 
assets 

Look  for: 

Clear  understanding  of  mission  objectives 

Awareness  of  what  were  the  critical  tasks  to  perform  in  order  to  accomplish  mission  objectives 

Awareness  of  how  mission  events  might  affect  accomplishment  of  mission  objectives 

Unnecessary  exposure  to  enemy  threats,  terrain,  and  close  proximity  of  other  aircraft  that  jeopardizes  the 
survivability  of  the  formation 

Ability  to  perform  self  analysis  of  progress  vs.  objectives  and  adjust  COA’s  accordingly 

1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

BO.  Overall  Mission  Execution  Behaviour 

Mission  execution 
behaviours  were 
significantly  below 
expectations. 

This  level  of 
proficiency  is  less 
than  desired  for 
effective 

coordination  during 
the  mission.  There 
is  room  for  much 
improvement. 

The  demonstrated 
behaviour 
promotes  and 
maintains 
coordination  and 
mission  operations 
effectiveness.  This 
is  the  minimum 
standard  level  of 
proficiency  that 
should  be 
expected  during 
this  mission. 

Behaviours  were 
significantly  above 
expectations. 

Behaviours 
represent  a  high 
level  of  skill  in  the 
application  of 
specific 

behaviours,  and 
serves  as  a  model 
for  coordination, 
teamwork  and 
highly  efficient 
mission  operations. 
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Mission  Debriefing 

Generic  Measures  of  Performance 

For  each  measure  listed  below,  please  circle  the  category  (i.e.  Poor,  Marginal,  Standard, 
Very  Good,  and  Exceptional)  that  maps  to  the  actions  you  just  observed.  Use  the 
behavioural  descriptions,  and  “Look  For’s”  to  direct  your  choice.  If  you  feel  the  measure  is 
Not  Applicable,  then  write  N/A  in  a  blank  space  somewhere  on  the  measure.  Please 
place  a  response  for  every  measure. 

1.  Poor  2.  Marginal  3.  Standard  4.  Very  Good  5. Exceptional 

Cl.  Overall  quality  and  communication 

No  debrief  Debrief  provided  All  debrief  Detailed  debrief  Highly  detailed 

provided  or  only  Critical  issues  left  elements  covered  with  all  mission  debrief  covering  all 

cursory  mission  .  H  with  acceptable  elements  covered  mission  elements 

debriefing  level  of  detail  for 

mission  objectives 

Look  for: 

Was  the  debrief  length  appropriate  to  the  complexity  of  the  mission  and  the  potential  lessons  learned 
Topics  covered  were  relevant  to  goals 
No  excessive  time  spent  on  any  unimportant  topics 

Audio-video/playback  tools:  did  crew  take  advantage  of  capabilities  to  review  their  own  technical/CRM 
performance? 

Were  audio/video  tools  cued  to  appropriate  occurrences  to  aid  in  de-brief  continuity. 

Does  the  debrief  follow  a  logical  review  of  mission  events? 

Was  the  reconstruction  of  the  mission  events  accurate? 

Did  the  mission  lead  involve  crew  when  required?  (Interaction  of  the  crew  during  reconstruction  and  debrief) 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

C2.  Flight  members  Team  Performance:  focus  on  team  performance/leadership 

Failed  to 
accomplish  a 
debrief  of  each 
fight  members’ 
team  performance 

Did  not  identify  any 
critical 

performance 

deficiencies 

Accomplished  a 
minimal  debrief  of 
each  fight 
members’  team 
performance 

Identified  few 
critical 

performance 

deficiencies 

Accomplished  a 
debrief  of  each 
fight  members’ 
team  performance 

Identified  only 
critical 

performance 

deficiencies 

Accomplished  a 
debrief  of  each 
fight  members’ 
team  performance 

Identified  critical  & 
some  non-critical 
performance 
deficiencies 

Accomplished  a 
thorough  and 
accurate  debrief  of 
each  fight 
members’  team 
performance 

Identified  all  critical 
&  non-critical 
performance 
deficiencies 

Look  for: 

Identified  and  expanded  on  areas  of  the  mission  where  team  performance/leadership  was  strong/weak. 
Recognized  lessons  learned  and  proposed  and  discussed  different  methods  to  improve  CRM. 

Were  critical  teamwork  incidents  covered? 

Is  ‘no  blame’  culture  maintained? 

Leadership,  followership  and  adherence  to  roles  and  responsibilities 

1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

C3.  Mission  outcomes:  focus  on  accomplishment  of  mission  objectives 

Failed  to 
accomplish  a 
debrief  of  mission 

Did  not  identify  any 
critical  mission 
deficiencies 

Accomplished  a 
minimal  debrief  of 
mission  objectives 

Identified  few 
mission  critical 
deficiencies 

Accomplished  an 
acceptable  debrief 
of  mission 
objectives 

Identified  only 
critical  mission 
deficiencies 

Accomplished  a 
debrief  of  mission 
objectives 

Identified  some 
mission 

deficiencies  and 
some  non-critical 
issues 

Accomplished  a 
thorough  and 
accurate  debrief  of 
mission  objectives 

Identified  all  critical 
and  some  non- 
critical  deficiencies 

Look  for: 

Does  debrief  centre  on  determining  whether  mission  objectives  were  achieved? 

Is  a  quantitative  assessment  conducted  to  assess  overall  mission  objective  success? 

Are  valid  reasons  for  failure  to  achieve  specific  mission  objectives  provided? 

Are  the  lessons  learned  recognized 

Focus  on  improving  performance 

Consideration  of  alternative  approaches  to  mission 

Do  flight  members  leave  debrief  with  positive  attitude?  Lessons  learned  if  mission  was  not  successful. 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

C4.  Flight  members  technical  performance:  focus  on  technical  performance 

Failed  to 
accomplish  a 
debrief  of  flight 
members’  technical 
performance 

Does  not  identify 
any  critical 
performance 
deficiencies 

Accomplished  a 
minimal  debrief  of 
flight  members’ 
technical 
performance 

Identified  few 
critical 

performance 

deficiencies 

Accomplished  a 
debrief  of  each 
flight  members’ 
technical 
performance 

Identified  only 
critical 

performance 

deficiencies 

Accomplished  a 
debrief  of  each 
flight  members’ 
technical 
performance 

Identified  critical 
and  some  non- 
critical 

performance 

deficiencies 

Accomplished  a 
thorough  and 
accurate  debrief  of 
each  flight 
members’  technical 
performance 

Identified  all  critical 
and  non-critical 
performance 
deficiencies 

Look  for: 


Was  there  a  distinction  between  team  technical  performance  and  flight  member  performance? 

Were  technical  errors  recognized  and  discussed  throughout  the  debrief? 

Is  the  debriefing  of  technical  performance  accurate  yet  efficient?  Were  ways  to  improve  provided  to  minimize 
errors  in  the  future? 

Does  the  debrief  include  discussion  of  all  relevant  events  that  influenced  the  outcome/success  of  mission? 
Were  concrete  examples  used  to  illustrate  points? 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

C5.  Technology  mediated  communication:  consider  the  effectiveness  of  devices  such 
as  Smartboards  and  video-conferencing,  during  distributed  meetings. 

Time  spent 
addressing 
technical  problems 
negatively  affected 
understanding 
mission  plan  or 
lessons  learned. 

Remote 

participants  did  not 
receive  the  same 
information  as  local 
participants. 

Interaction  with 
devices  distracted 
from  the  goal  of  the 
meeting. 

Some  time  was 
spent  addressing 
technical 
difficulties. 

Had  to  wait  for 
remote  participants 
to  receive 
information. 

Poor  transitions 
between  briefings 
due  to  technical 
difficulties. 

Some  time  was 
spent  addressing 
technical 
difficulties. 

Local  and  remote 
participants 
received  the  same 
information. 

Incorporates  some 
devices  into 
mission  execution 
plan. 

The  use  of  the 
devices  did  not 
distract  from  the 
meeting. 

Similar  experience 
as  an  “in  person” 
meeting. 

Integrates  many 
additional  devices 
into  the  mission 
execution  plan. 

Remote 

participants  had 
same  level  of 
input/interaction  as 
local  participants. 

Devices  were 
seamless  used  in 
the  presentation. 

Devices  used  in 
the  meeting  was 
optimal  for  the 
presentation  of  the 
mission  execution 
plan 

Look  for: 

Could  you  identify  all  air  crew  participating  at  the  remote  sites? 

Could  you  tell  if  remote  people  could  be  interrupted? 

Could  you  tell  when  remote  participants  were  ready  to  continue  to  the  next  topic  of  the  briefing? 

Was  there  a  need  for  private  communication  with  participants  at  a  remote  site? 

Was  the  transition  between  briefings  seamlessly  accomplished? 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5. Exceptional 

CO.  Overall  Debriefing  Behaviour 

Observed 
debriefing 
behaviours  were 
significantly  below 
expectations. 

This  level  of 
proficiency  is  less 
than  desired  for 
effective 

coordination  during 
the  mission.  There 
is  room  for  much 
improvement. 

The  demonstrated 
behaviour 
promoted  and 
maintained 
coordination  and 
mission  operations 
effectiveness.  This 
is  the  minimum 
standard  level  of 
proficiency  that 
should  be 
expected  during 
this  mission. 

Observed 
debriefing 
behaviours  were 
significantly  above 
expectations. 

Behaviours 
represent  a  high 
level  of  skill  in  the 
application  of 
specific 

behaviours,  and 
serves  as  a  model 
for  coordination, 
teamwork  and 
highly  efficient 
mission  operations. 
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There  are  two  Excel  data  files  supporting  this  report:  CSCW  data  tables  are  located  using  this  file 
path: 

CSCW  Data  Tables 

CSCW  detailed  phase  data  can  be  located  in  the  individual  mission  excel  files  using  this  path: 
CSCW  Phase  Data 


The  CSCW  Data  Tables  includes  several  individual  sheets.  These  sheets  are  described  below. 

51.  TurnOperator 

Session 

INTB  =  initial  brief 

MB  =  mass  brief 

MD  =  mass  debrief 

MIDCORD  =  mid  coordination  brief 

FINALCORD  =  final  coordination  brief 

STMD  =  Strikers  and  Transport  mass  debrief 

Participant 

P  with  a  number  or  letter  identifies  a  participant.  P7  represents  technical  support  and  P 1 0 
represents  all  participants  laughing. 

Turn  Type 

S  =  speech 

C  =  computer  use  (primarily  power  point  operator  facilitating  presentation  display) 
Overlap 
O  =  overlap 

An  overlap  identifies  when  a  participant  who  is  engaged  in  a  speaking  turn  is  talked  over 
by  another  participant 

Turn  Length 

Number  of  seconds  a  participant’s  turn  lasts.  Speaking  turn  is  defined  when  a  participant 
initiates  a  verbal  utterance  followed  by  silence. 

Overlapped  By 

Overlapped  By  identifies  what  participant  commits  an  overlap  against  another  participant. 

52.  Transition  Count 

C  =  computer  use  (action  on  operator  computer) 
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I  =  introduction  (when  presenter  introduces  next  presentation  speaker) 

Q  =  question  (question  asked  by  participant) 

RC  =  role  call  (name  called  out  and  participant  answers  “here”)**  not  used  in  report 
T  =  thank  you  (presentation  moderator  sometimes  thanked  individual  presenters) 

W  =  welcome  (incorporated  into  I  for  reporting  purposes) 

53.  Transition 

Refer  to  spreadsheet  “TumOperator”  and  “Transition  Count”  for  definitions 

54.  Active  Participant  Pivot 

Refer  to  spreadsheet  “Tum  Operator”  and  “Transition  Count”  for  definitions 

55.  Active  Participants 

Room 

1  =  upstairs  SERF  room 

2  =  downstairs  SERF  room 

56.  Tool  Frequency  Count 

C  =  computer 
P  =  Polycom  system 
PR  =  printer 
SB  =  SMARTboard 

57.  Breakdowns 

Breakdown  Type 
U  =  user  error 

LOE  =  lack  of  operating  experience 
S  =  software  malfunction/issue 

58.  Percentile 

Denotes  the  average  ‘short’  turn,  the  average  ‘medium’  turn,  and  the  average  ‘long’  turn, 
as  well  as  the  maximum  and  minimum  turn  lengths  and  the  most  common  duration  of  turn. 

59.  Overlap 


FI  umansys  terns’  Incorporated 


CSCW  Tool  Usage 


B  -2 


HUMANSYSTE.US 

UcorpofUi 


A  pivot  table  describing  who  was  talked  over  by  whom,  how  many  times.  Refer  to  “Turn 
Operator”  for  definitions. 


S10.  Number  of  Turns 

A  pivot  table  describing  the  number  turns  each  participant  had  during  briefings  for  each 
mission.  Refer  to  “Turn  Operator”  for  definitions. 


Sll.  Tools  Freq  Pivot 

A  pivot  table  describing  the  number  of  times  each  tool  was  used  during  briefings  for  each 
mission.  Refer  to  “Tool  Frequency  Count”  for  definitions. 


S12.  Tools 

Raw  data  regarding  tool  usage.  Refer  to  other  sheets  for  definitions. 


S13.  Number  of  Breakdowns 

A  pivot  table  describing  the  number  of  breakdowns  in  the  briefings  for  each  mission. 


The  CSCW  Phase  Data  contains  the  raw  data  describing  each  participants’  turns.  The  first  column 
(Participant)  denotes  the  participant  (use  “Turn  Operator”  for  definitions),  and  the  second  and  third 
columns  denote  the  start  and  end  of  that  person’s  turn. 
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